Nodes
A model node is any device running the gateway app with "role": "node". It serves its models to the hub only.
- Outbound only. The node opens one WebSocket to
wss://<hub-host>/v1/nodes/connectwith its node key. With"listen": false(the default for WebSocket nodes) nothing listens on the device at all. - Hello and heartbeat. The first frame lists the node’s models and its device: manufacturer, model, SoC, OS, RAM and accelerators. A heartbeat every 30 s carries model states and queue depth.
- Jobs. Each
rungoes through the node’s normal path: body limits, image decode, the model’s bounded queue (429busywhen full), load, timeout, metrics and the node’s own audit log. - Scheduler. One worker per model with a bounded queue, lazy load on first use, idle unload after 10 minutes, and a memory budget that unloads the least-recently-used idle models to fit.
- Accelerators. The manifest lists accelerators in order of preference; the node keeps the first runner that
loads and reports it as
backend. - Benchmarks. After five runs of a model, the node reports p50/p95 runner times and a status to the registry.
- Reconnect. Exponential backoff with jitter (1 s → 60 s), paused without a network, immediate on a network change.
Same-device apps on a phone node keep using Binder IPC to the phone’s own models, whether or not the hub link is up.
The wire format is in the node protocol.