Skip to content

Nodes

A model node is any device running the gateway app with "role": "node". It serves its models to the hub only.

  • Outbound only. The node opens one WebSocket to wss://<hub-host>/v1/nodes/connect with its node key. With "listen": false (the default for WebSocket nodes) nothing listens on the device at all.
  • Hello and heartbeat. The first frame lists the node’s models and its device: manufacturer, model, SoC, OS, RAM and accelerators. A heartbeat every 30 s carries model states and queue depth.
  • Jobs. Each run goes through the node’s normal path: body limits, image decode, the model’s bounded queue (429 busy when full), load, timeout, metrics and the node’s own audit log.
  • Scheduler. One worker per model with a bounded queue, lazy load on first use, idle unload after 10 minutes, and a memory budget that unloads the least-recently-used idle models to fit.
  • Accelerators. The manifest lists accelerators in order of preference; the node keeps the first runner that loads and reports it as backend.
  • Benchmarks. After five runs of a model, the node reports p50/p95 runner times and a status to the registry.
  • Reconnect. Exponential backoff with jitter (1 s → 60 s), paused without a network, immediate on a network change.

Same-device apps on a phone node keep using Binder IPC to the phone’s own models, whether or not the hub link is up.

The wire format is in the node protocol.