Node protocol
A model node (the phone, the TV) connects outbound to the hub (usually a server behind a CDN
and a reverse proxy) over one persistent WebSocket and receives jobs on it. Nodes may be behind NAT or on
mobile data; they open no inbound port. This is the default node transport ("hub": {"transport": "websocket"});
the older HTTPS self-registration ("transport": "https", the hub calls the node back) is described in the README.
Implemented by the Android node in app/src/main/java/dev/kinherd/app/node/, the Linux node in
node-linux/internal/hub (node-linux/README.md), and by the hub in
server/internal/protocol and server/internal/nodes.
Connection
Section titled “Connection”GET wss://<hub-host>/v1/nodes/connectAuthorization: Bearer <node token>- TLS: the system trust store with host-name verification (a public certificate, e.g. through Cloudflare). For a
self-hosted hub with a self-signed certificate the node may pin the leaf instead (
"certSha256", SHA-256 of the DER); then only that certificate is accepted and the host name is not checked. - Auth: the node token identifies the node; the hub names the node after it (the
nameinhellois a label). A refused upgrade (HTTP 401/403) makes the node retry at the slowest backoff. - Keep-alive: both sides send WebSocket pings (the node every 20 s), so dead connections (NAT timeouts, a lost Wi-Fi) are noticed within about 20–40 s.
- Frames: JSON text frames, each an object with a
"type". Binary frames are not used. Unknown types and unknown fields are ignored by both sides; fields may be added (optional) but never renamed or removed.
Messages
Section titled “Messages”node → hub hello (first frame)
Section titled “node → hub hello (first frame)”{"type": "hello", "name": "pixel-7-pro", "appVersion": "0.3-litert2", "models": [{"id": "clip-l14-336", "version": "1", "task": "image-embedding", "backend": "litert-gpu", "state": "loaded"}], "device": {"manufacturer": "Google", "model": "Pixel 7 Pro", "soc": "Google Tensor G2", "os": "Android 16 (API 36)", "ramMB": 11683, "accelerators": ["gpu", "npu", "cpu"]}}models[]:id,version,task,backend(null until the model has loaded),state(unloaded|loading|loaded|error). Optional extras:name,memoryMB,input,output.device:socisBuild.SOC_MANUFACTURER + SOC_MODEL(API 31+), elsero.board.platform, elseBuild.HARDWARE.acceleratorsare those of the build flavor’s runtime (litert2: gpu, npu, cpu;nnapi: npu, cpu). Optional extras, used to match recipes (RECIPES.md):abi(the primary ABI,Build.SUPPORTED_ABIS[0], or the whole list as an array),osVersion(Build.VERSION.RELEASE, e.g."14"),sdkInt(Build.VERSION.SDK_INT),cpus. The Linux node sendsmanufacturer/modelfrom DMI or the device tree (Generic/Linux <arch>when there is none),soc= the device-tree SoC and/or the CPU model,os="Linux <kernel> (<distro>)",osVersion= the kernel version,abi=["aarch64","arm64"]or["x86_64","amd64"],accelerators: ["cpu"](DEVICE_PROBE.md, Hello device facts).probe(optional): the node’s full device probe, DEVICE_PROBE.md (probeVersion, OS, CPU and ISA extensions, memory, GPUs/NPUs, usable compute APIs, power, thermal, micro-benchmarks). Sent by the Linux node inhelloonly (not in heartbeats); a few KB. It carries no host name, user, serial numbers or addresses. The hub validates and sanitises it (≤ 64 KB,probeVersion1, unknown fields dropped), keeps the latest per node for its estimator and the anonymous registry aggregate (DEVICE_PROBE.md, Hub); a bad probe is logged and ignored, never a reason to refuse the hello. Older hubs ignore the field.devicestays the matching fingerprint, the probe is extra detail.direct(optional): the node’s LAN HTTPS endpoints for direct connections,{"endpoints":["https://<lan-ip>:9443"],"certSha256":"<64 hex of the DER cert>","subnets":["<lan-ip>/24"]}(DIRECT.md).protections(optional): the owner protections this node enforces and whether it serves now, see Owner protections below. Older hubs ignore it; a hub with the marketplace on offers a node’s listings to other owners only when its hello carried this object.features(optional): protocol extensions the node supports, e.g.["chat-stream","cancel"](see Chat models and streaming below). A hub uses a feature only with nodes that list it; older hubs ignore the field and older nodes simply do not send it.jobKey(optional): the node’s current job key for end-to-end sealing, signed by its device key ({"v":1,"kid","pub","iat","exp","device":{"alg","pub"},"sig"}, E2E.md §2). The hub verifies it and ignores an invalid one; only nodes with a valid job key get sealed runs. Older hubs ignore it. The welcome then carriessealingNonce; the node answers{"type":"jobKeyProof","kid","sig"}(its device key’s signature over the welcome’s hub name, the nonce and the kid, E2E.md §2) and the hub replies{"type":"jobKeyStatus","kid","status":"active"|"refused","reason"}; only an active job key is routed to.
hub → node welcome
Section titled “hub → node welcome”{"type": "welcome", "hub": "home-hub", "heartbeatSec": 30, "node": "phone", "maxInFlight": 4}heartbeatSec is clamped by the node to 5–300 (default 30). Optional: node (the name the hub registered the node
under, shown on the node’s status screen) and maxInFlight (the node then runs at most that many jobs at once).
Optional pending: true: the node was paired but an admin has not approved it (PAIRING.md); it gets
no runs and no recipes until the hub sends {"type": "approval", "status": "approved"}.
Optional tickets ({"keyId","publicKey":"<base64>"}, the key that signs direct-connection tickets now) and
ticketKeys (every ticket key to accept, active first); absent when direct connections are off (DIRECT.md).
Optional lan ([{"url":"wss://<lan-ip>:8443/v1/nodes/connect","certSha256":"<64 hex>"}], at most 4): the
hub’s LAN endpoints, see LAN-first below. Absent = the hub advertises none (a node then forgets any it saved).
A node that gets no welcome within 20 s closes and reconnects. Runs that arrive before the welcome get a 503 result.
node → hub heartbeat (every heartbeatSec)
Section titled “node → hub heartbeat (every heartbeatSec)”{"type": "heartbeat", "models": [ … as in hello … ], "queue": 0}A node that pauses (the Linux node on battery or while its user is active) sends a heartbeat with "models": []
at once, so the hub stops routing to it without dropping the connection, and the full list again when it resumes;
a run that still arrives while paused gets 503 unavailable with retryAfter.
queue: requests waiting in the node’s model queues (all callers, including same-device apps).
Optional direct as in hello: missing = unchanged, {} or empty endpoints = withdrawn.
Optional protections as in hello: missing = unchanged (both nodes send it in every heartbeat).
Optional jobKey as in hello: missing = unchanged; a node sends it when it rotated its job key.
hub → node run
Section titled “hub → node run”{"type": "run", "id": "<request id>", "model": "clip-l14-336", "contentType": "image/jpeg", "bodyB64": "<base64 image>"}id: 1–128 characters (a number is accepted too). bodyB64: standard base64 (line breaks tolerated).
Optional path (hubs with the path indicator): how the request reached this node, "hub-lan" (the client reached
the hub over the LAN and this node’s link is on the LAN, e.g. through the LAN listener) or "hub-public" (either
half used the public URL / Cloudflare). Purely informational, for a “Local ✓ / Via cloud” display; nodes ignore
unknown values (SERVER.md). A link through the LAN listener is what
makes relayed jobs eligible for local only keys.
Optional stream: true: only to nodes whose hello listed chat-stream, for a chat model; the node answers with
chunk frames and then the usual result (Chat models and streaming).
Sealed runs (E2E.md): contentType application/aigw-sealed means bodyB64 is a sealed request
envelope addressed to the node’s job key (the hub sends these only to the node whose advertised kid it names).
The node opens it with the binding (its own name, model, empty jti), checks freshness and replay, runs the inner
body as usual, and answers body: {"sealed": "<base64 response envelope>"}; on an error after opening it answers
the status with {"error":{"code":"<code>","message":"sealed"},"sealed":"…"} (the full error object sealed).
Envelope problems before opening are plain errors: 400 sealed_invalid / sealed_stale, 409 sealed_replay /
job_key_unknown. usage stays outside the ciphertext.
node → hub result (one per run)
Section titled “node → hub result (one per run)”{"type": "result", "id": "<request id>", "status": 200, "body": { … native run JSON … }}{"type": "result", "id": "<request id>", "status": 429, "body": {"error": {"code": "busy", "message": "…"}}, "retryAfter": 1}body is exactly what POST /v1/models/{id}/run on the node would answer: on success
{"model":{"id","version"},"output":{…},"ms","backend","servedBy","timings"}, otherwise the error object with the
same status and code as the HTTP API (native API). Optional retryAfter (seconds) accompanies 429 and 503.
Optional usage reports the tokens the run consumed, for models that count them (text models; vision models
simply omit it):
{"type": "result", "id": "<request id>", "status": 200, "body": { … }, "usage": {"input_tokens": 12, "output_tokens": 0}}Both fields are non-negative integers (at most 100 000 000; other values are ignored). The hub counts them against
the key’s and team’s tpmLimit and returns them to the client (usage on the run answer, usage.prompt_tokens /
total_tokens on /openai/v1/embeddings, the full OpenAI usage on /openai/v1/chat/completions). Hubs that predate the field ignore it, and nodes that do not send it
keep working unchanged. HTTPS-registered nodes may put the same object at the top level of their answer body.
The job goes through the node’s normal API path: model id check (404 model_not_found), body size limit
(413 payload_too_large, empty body 411), image decode (415), the model’s scheduler queue (429 busy when full),
load failures (503 unavailable), requestTimeoutSeconds (504), metrics and the audit log (client = the hub’s name
from welcome, ip hub-link, method WS). The node runs several jobs concurrently, up to
models × (queueSize + 1) (capped at 64, and at the welcome’s maxInFlight); beyond that it answers 429 busy at
once. A malformed run whose id is readable gets a 400 bad_request result; without an id it is dropped. Results
for a connection that has since dropped are discarded (the hub has already failed those requests).
Chat models and streaming
Section titled “Chat models and streaming”A node advertises a chat (LLM) model with "task": "chat" in models[], like any other model:
{"id": "qwen2.5-1.5b-instruct", "version": "q4_k_m", "task": "chat", "backend": "llama.cpp-metal", "state": "loaded"}The hub lists it on GET /openai/v1/models and routes POST /openai/v1/chat/completions for it to the node as a
run with contentType: "application/json" and, as the body, the client’s OpenAI chat completion request
(model, messages, temperature, tools, … unchanged) without stream and stream_options, and with
max_tokens (or max_completion_tokens) capped at the hub’s openai.maxTokens (set when the client sent none).
Tool calling is passed through: tools (at most 128, 256 KiB, checked by the hub), tool_choice,
parallel_tool_calls, assistant tool_calls and tool messages arrive as the client sent them; the answer’s
message.tool_calls / streamed delta.tool_calls (with index) and finish_reason: "tool_calls" go back unchanged
(LLM.md).
A node must not offer a model id that is one of the hub’s OpenAI aliases (openai.models); the hub ignores it.
Images are only inline image_url data URLs (data:image/…;base64,…); the hub refuses remote URLs, so a node never
has to fetch anything.
The result body is an OpenAI chat.completion object (id, created, choices[] with message and
finish_reason, usage with prompt_tokens / completion_tokens); the node should also put the counts in the
result’s usage (input_tokens = prompt, output_tokens = completion). The hub counts them against the key’s and
team’s token limits and returns them in OpenAI’s usage. Errors are the usual error results (model_not_found,
busy, unavailable, timeout, …).
Streaming (feature chat-stream). When the client asks for "stream": true and the node listed chat-stream,
the run carries "stream": true and the node sends any number of
{"type": "chunk", "id": "<request id>", "data": {"id": "chatcmpl-…", "object": "chat.completion.chunk", "created": 1767225605, "model": "qwen2.5-1.5b-instruct", "choices": [{"index": 0, "delta": {"content": "Hel"}, "finish_reason": null}]}}in order, then the result as above (its body may be the assembled completion or just {"object":"chat.completion", "usage":{…}}; the hub uses only its status and usage). data is one OpenAI chunk object; chunks without choices
(e.g. a usage-only chunk) are dropped, and the hub sends its own usage chunk when the client asked for
stream_options.include_usage. A result with an error status before any chunk is a normal error answer; after
chunks it ends the client’s stream with an error event. The hub buffers at most 8 MiB of chunks per run for a slow
client, then cancels the run.
A node without chat-stream gets stream never set and answers in one piece; the hub turns that answer into
server-sent events itself, so clients can always stream.
Cancel (feature cancel). When the client goes away (or the hub’s request timeout passes) the hub sends
{"type": "cancel", "id": "<request id>"} to nodes that listed cancel; the node stops generating and should
still send its result (any status, e.g. 499) with the usage spent so far: the hub waits up to 2 s for it and
counts the larger of that report and its own estimate (also used when a node reports no usage at all, or a run is
lost midway).
Disconnects. A link that drops before the first chunk is a transport failure like any other (the hub retries
another node with the model). After a chunk was relayed the hub does not retry (the answer would start over): the
client’s stream ends with an error event (peer_unavailable) and no data: [DONE].
node → hub benchmark
Section titled “node → hub benchmark”{"type": "benchmark", "results": [ {"model": "clip-l14-336", "version": "1", "backend": "litert-gpu", "status": "works", "p50ms": 1412.3, "p95ms": 1530.8, "notes": "", "samples": 64}, {"model": "efficientnet-lite4", "version": "1", "backend": "xnnpack-cpu", "status": "tweaks", "p50ms": 95.1, "p95ms": 110.2, "notes": "fell back, skipped npu: …"}, {"model": "yolo", "version": "2", "backend": null, "status": "fails", "p50ms": null, "p95ms": null, "notes": "no accelerator could load yolo (…)"}]}p50ms/p95ms: nearest-rank percentiles of the scheduler’s last 64 runner times on that backend (render, pack, invoke, postprocess: no queueing or load), 0.1 ms resolution.status:works= loaded on the manifest’s first-choice accelerator;tweaks= it loaded only after falling back (noteslists the skipped accelerators and why);fails= the model failed to load (noteshas the error).- When: after a model has 5 runs on its backend (so after its first successful loads), and for a load failure;
checked after the welcome and after every heartbeat. At most every 6 h per model, except straight after a model
goes from
failsto working. Optional extra:samples. - On demand: the hub may send
{"type": "benchmark"}(hub → node, optional); the node answers with every model that has at least one run or a load failure, ignoring the 6 h limit.
hub → node recipes
Section titled “hub → node recipes”{"type": "recipes", "recipes": [{"schema": 1, "id": "tcl-smart-tv-pro-mt9689", "version": 1, …, "signature": {"keyId": "…", "sig": "…"}}]}The signed recipes that match the node’s device, best match first (possibly empty), sent after the welcome and
again whenever the hub’s recipes change. Each element is the recipe document exactly as signed (canonical JSON);
the node verifies it with its trusted publisher keys. Format and apply steps: RECIPES.md.
hub → node trust
Section titled “hub → node trust”{"type": "trust", "updates": [{"type": "trustUpdate", "schema": 1, "serial": 1, "prev": 0, …, "signatures": [{"keyId": "…", "sig": "…"}]}, …]}The hub’s chain of signed publisher trust updates, oldest first, as stored (canonical JSON). The hub sends it right
after the welcome, before recipes (so recipes signed by a key the chain adds verify), and again to every
connected node when an admin uploads an update. A hub without updates sends no trust frame; nodes that do not
know the frame ignore it. Pending nodes get it too: it carries no work, only keys.
The node applies, in serial order, each update newer than the serial it has applied, under the rules in
RECIPES.md (Rotating the publisher key with a trust update): prev chained to the applied serial (any
prev for a node that never applied one), inside notBefore/notAfter, signed by enough keys the node trusted
before the update. It saves the new key set and serial together (atomically) before trusting them, logs every
applied or refused update, and sends nothing back. Updates it has already applied are skipped silently; a frame
with nothing new is a no-op. The same list is in the pairing answers (trustUpdates) and at GET /v1/trust
(node key).
node → hub recipeStatus
Section titled “node → hub recipeStatus”{"type": "recipeStatus", "recipe": "tcl-smart-tv-pro-mt9689", "version": 1, "status": "applied", "notes": "p50 invoke 31 ms"}status: applied (installed, configured, and the verify benchmark met its expectations), failed, or skipped
(the node chose not to apply it). Send the usual benchmark frame as well.
hub → node approval
Section titled “hub → node approval”{"type": "approval", "status": "approved"}: a pending node was approved; runs and recipes follow.
hub → node tickets (optional)
Section titled “hub → node tickets (optional)”{"type": "tickets", "tickets": {"keyId", "publicKey"}, "ticketKeys": [ … ]}: the ticket keys changed (an admin
rotated them); replace the welcome’s keys. A node that ignores it gets the new keys on its next welcome.
node → hub directUsage
Section titled “node → hub directUsage”{"type": "directUsage", "items": [{"jti": "…", "seq": 1, "sub": "7", "model": "clip-l14-336", "status": 200, "ms": 41, "bytes": 182344, "at": 1767225605, "receipt": "1.4VfiqOYEaknkKBZJkN5NcglGT5Of61SVvkbdtSyGXn_zRk0_cJdtCaDrNTDGwg2tDhQdnmV1tECxMuOyBlCzAw", "bodySha256": "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"}]}Requests the node served directly for hub tickets, batched (at most 512 items per frame; at in unix seconds).
seq is the use number of the ticket (1 … uses; 1 for single-use tickets). receipt is the client’s
X-Use-Receipt request header, copied verbatim (omit when the request had none), and bodySha256 the lower-case
hex SHA-256 of the request body the node received; the node verifies the receipt against the ticket’s rpk
before serving (401 invalid_receipt otherwise) and the hub charges a priced use only when it verifies.
Each (jti, seq) is recorded once, so a batch may be resent after a reconnect. An item may carry the same optional
usage object as a result; its tokens count against the key’s token limits. Details: DIRECT.md §5.
Close codes (hub)
Section titled “Close codes (hub)”| Code | Meaning | Node reaction |
|---|---|---|
| 1001 | hub shutting down | reconnect with backoff |
| 4400 | first frame was not a valid hello | slowest backoff (60 s) |
| 4408 | no traffic for the hub’s offline period | reconnect with backoff |
| 4409 | the same node connected again; this connection is superseded | slowest backoff |
| 4429 | node limit reached | slowest backoff |
| 4401 | the node token was revoked (also mid-handshake) | slowest backoff; a new token is needed |
Reconnect
Section titled “Reconnect”Exponential backoff with equal jitter: attempt n waits a uniform time between half and all of
min(60 s, 1 s · 2ⁿ), never under 1 s (1 s, 1–2 s, 2–4 s, … 30–60 s), so nodes do not reconnect in lock-step after a
hub restart. The counter resets on welcome. The node pauses while the device has no network (no attempts at all)
and reconnects at once, with the counter reset, when the default network returns or changes (Wi-Fi ↔ mobile data);
a socket on the old network would only time out. After an app restart or reboot the node reconnects when the
service starts (it remembers Start/Stop).
Owner protections
Section titled “Owner protections”A node pauses serving whenever its owner’s rules are not met, so lending the device never costs its owner battery,
responsiveness, heat or mobile data. Paused, it stays connected, sends a heartbeat with "models": [] at once (and
the full list on resume), withdraws its direct advertisement ("direct": {}), and answers any run (relayed or
direct) with 503 unavailable, retryAfter 60.
"protections": {"requireCharging": true, "requireIdle": true, "idleMinutes": 5, "screenOff": true, "maxThermal": "moderate", "allowMetered": false, "serving": false, "reason": "on battery; screen on"}| Field | Android (Settings → Owner protections) | Linux (config.json policy) |
|---|---|---|
requireCharging |
only while plugged in (BatteryManager; no battery = always plugged) | requireAC |
requireIdle, idleMinutes |
only after the screen has been off this long (screenOff: true; PowerManager.isInteractive, ACTION_SCREEN_OFF) |
requireIdle, idleMinutes (no input, no CPU load by others) |
maxThermal |
pauses at this PowerManager thermal status or worse (light … severe; omitted = never) |
— |
maxTempC |
— | maxTempC |
allowMetered |
false: pauses while the default network is metered (mobile data, metered hotspot) | always true (not detected) |
serving, reason |
the current decision; reason only while paused |
same |
Android defaults: phone / tablet charging only, screen off for 5 min, pause from thermal moderate, no
mobile data; TV / no battery (always plugged, and serving while people watch is the point) only thermal
severe and no metered data. Before the first check a node reports serving: false (“checking the device”).
The dashboard shows the state on the Owner protections card; the hub shows it on Nodes and in GET /v1/nodes
(protections, null when the node reports none).
LAN-first
Section titled “LAN-first”So that home inference keeps working when the internet or Cloudflare is down, a node dials the hub’s LAN endpoints
before the public URL it was paired with. It is zero-config: a normal install (server/docker-compose.yml, a
Swarm stack, or the binary on a host) needs no address, port or firewall setting.
- The hub’s LAN listener is on by default: a second, TLS-only listener that serves only
/healthz//livez//readyz, the pairing endpoints and node-key paths (no dashboard, no client API). Port 8443, else the port it used last time or the next free one (8444–8453, then any), remembered in<dataDir>/lan-tls/port. Its certificate is self-signed and generated once in<dataDir>/lan-tls/(nodes pin it, no CA needed). In the default (auto) mode it binds only the hub’s private LAN addresses, never the wildcard, so a hub that also has a public interface does not expose it; inside a bridge/overlay container it binds the container’s wildcard (the host publishes the port). Whatever it binds, it closes connections from peers that are not private (RFC 1918, ULA), loopback or link-local, and it ignoresX-Forwarded-For/CF-Connecting-IP(nothing proxies it). Pairing through it has a global cap (30 code redemptions, 30 device-flow starts, 1200 polls per 10 min) on top of the per-IP limit. - Address detection. The hub advertises
wss://<addr>:<port>/v1/nodes/connectfor each private address of its interfaces: IPv4 RFC 1918 first, then IPv6 ULA (fc00::/7); loopback, link-local (169.254/16, fe80::/10: no stable or zone-free address), CGNAT/VPN (100.64/10) and public addresses are skipped, as are interfaces of container/VM/ VPN plumbing (docker*,br-*,veth*,virbr*,cni*,flannel*,cali*,vxlan*,tun*/tap*,wg*,tailscale*,zt*, …; a Proxmoxvmbr0or a NASbr0carrying the LAN address is kept). At most 4, re-detected every minute (DHCP changes reach nodes with their next welcome and mDNS at once; new addresses get a socket).- Host, or a container with host networking (
network_mode: host, the shipped compose file): the hub sees the host’s interfaces. - Container with bridge/overlay networking (
/.dockerenvpresent and no host-only interface such asdocker0visible): its own addresses are useless to the LAN, so it detects none and relies onAIGW_HOST_ADDRS(addresses or names, optional:portwhen the published port differs). A Swarm stack publishes 8443 in host mode; the deployment fillsAIGW_HOST_ADDRSby running the image’s/server -lan-addrswith host networking on the target host (same detection rules), and the stack also passes Swarm’s{{.Node.Hostname}}asAIGW_HOST_NAME, resolved best effort. The Docker host gateway (172.17.0.1, host.docker.internal) is no LAN address, so it is not used. - Overrides, all optional:
AIGW_LAN_URLSreplaces the detected list,AIGW_LAN_LISTEN=offdisables the listener,AIGW_LAN_LISTEN=:9443fixes the port (binds what it says, also the wildcard; no fallback).
- Host, or a container with host networking (
- mDNS / DNS-SD. The hub announces itself as
_aigw._tcp(instance = hub name; SRV = LAN port, A/AAAA = the detected addresses; TXTv=1,name,fp= the LAN certificate’s SHA-256,id= its first 16 hex digits,pub= public URL,port,api/apitls= main listener,ver). No secrets.AIGW_MDNS=offdisables it. It only reaches the hub’s own network segment (multicast does not cross VLANs or leave a bridge-networked container). - Discovery at pairing. Without a hub address,
kinherd-node pairand the app’s Find the hub on this network list the announced hubs, full fingerprint first (all 64 hex digits in groups of four, the value the dashboard’s Add device page shows), then the addresses, then the name (plain text; names with control or bidi characters, or over 63 bytes, are dropped). The user compares the whole fingerprint and confirms, or passes--fingerprint <full SHA-256>; a pin already given (--cert-sha256, a payload’shubCertSha256) must match. Pairing then goes to the hub’s LAN listener over TLS pinned to that certificate (code or device flow as usual), never to the announcedpubURL or plain http: an impostor announcing the real fingerprint cannot complete the handshake. The answer, authenticated by the pin, may name the hub’s publicwss://URL as the link (then checked against the system trust store); the LAN endpoints are saved pinned. Discovery never pairs or trusts by itself: the pairing code or the admin’s approval still decides. - Learning the list. The hub lists its LAN endpoints in every welcome (
lan). The node keeps the last list (Android:hub-lan.jsonin app storage, bound to the paired hub URL; Linux:lanincredentials.json; both are dropped by a new pairing), so a start with the internet down still finds the hub. - Order and fallback. Each round tries every LAN endpoint in order, then the public URL. A dial that fails before the welcome (refused, unreachable, TLS pin mismatch, no welcome within 5 s) moves to the next candidate at once; only when every candidate failed does the normal backoff apply, and the next round starts again with the LAN. A 401/403 or a slow close code (4400/4401/4409/4429) ends the round at once (the public URL would answer the same).
- Stale list. When every remembered LAN endpoint failed, the node asks mDNS (2–3 s, at most once a minute;
AIGW_MDNS=offon the Linux node disables it) before the public URL, and dials only answers pinned to a certificate the hub already advertised over the authenticated link (a hub that moved to another address or port). With no known pin nothing found is dialled. - Failback. While connected over the public URL, every 2 min (doubling after each round in which no LAN
endpoint answered, up to 30 min; a new LAN list or a switch starts at 2 min again) the node checks
GET https://<lan>/healthz(TLS against the pin, no token). When a LAN endpoint answers 200 and no job is running, the node closes the public connection (1000) and reconnects, LAN first. A job in flight postpones the switch to the next check. - Security. LAN endpoints must be
wss://;ws://entries are ignored by nodes and refused by the hub’s config, so the node token never crosses the LAN in clear. WithcertSha256the node accepts exactly that leaf certificate; without it, the system trust store and host-name check apply. The pin arrives over an authenticated link (the public link, or the pinned discovery pairing), so someone on a café Wi-Fi that happens to use the same address gets a TLS handshake failure and never the token. Package/recipe downloads still use the public URL. - Status. Android shows “connected to home-hub (LAN)” and the endpoint in use;
kinherd-node statusshows(lan, wss://…)or(public, …). The app logs each NSD browse under the logcat tagaigw-mdns.
Segmented networks / VLANs (outside the normal flow)
Section titled “Segmented networks / VLANs (outside the normal flow)”When nodes live on another VLAN or subnet than the hub (an IoT VLAN, a guest network):
- Allow TCP
(default 8443) from the node VLAN to the hub’s address on the router/firewall (for example a pass rule on the node VLAN’s interface, destination the hub, port 8443). Without it nodes simply use the public URL. - mDNS does not cross VLANs. Either run an mDNS reflector (the router’s Avahi package with reflection between the two
interfaces, or
avahi-daemonwithenable-reflector=yes), or pair with the hub address / QR code instead of discovery. Already paired nodes do not need mDNS: the hub’s welcome tells them its LAN endpoints, which work across VLANs once the port is allowed. - The public URL keeps working as the fallback either way.
Hub side (server edition)
Section titled “Hub side (server edition)”What server/ does with this protocol (deployment: Run a hub):
- URLs:
wss://hub.example.com/v1/nodes/connectfrom anywhere, orws://<hub LAN address>:8080/v1/nodes/connecton the LAN. Node tokens are virtual keys with"kind": "node"(dashboard Keys, orPOST /v1/admin/keys {"name":"tv","kind":"node"}); the key’s name becomes the node’s name. Before the upgrade the hub answers 401 (bad key), 403 (not a node key) or 429 in the usual error schema. - Limits: the hello must arrive within 15 s; frames from a node up to 16 MiB (
runframes the hub sends are at most ~13.4 MiB, a 10 MB image in base64, so a node’s read limit must be at least that); no compression. - Liveness: the hub pings every 20 s and closes with 4408 after 90 s without frames or pongs.
- Routing: among online nodes with the model:
stateloadedfirst, then the lowest EWMA round-trip latency for that model, then the fewest jobs in flight; at mostmaxInFlight(default 2) jobs per node, further requests wait up to 10 s for a slot, then get 429busy. If the connection drops with jobs in flight they are retried on another node with the model, else the client gets 503. Only transport failures are retried; a node’s error answer or a hub timeout (requestSec, 504) is final. - Answers: a 200
bodyis passed to the client withservedBy= the node’s name andvia= the hub’s name. Errors pass through unchanged except 401/403 and non-object bodies (502peer_unavailable);model_not_founddrops that model from the node until its next heartbeat.retryAfterbecomesRetry-After. - Benchmarks go into the compatibility registry (
GET /v1/registry) with the hello’sdevice,source: "auto"and the node’s name; a newer automatic result for the same model, version, device and backend replaces the older one.manufacturerandmodelindeviceare required for this. The estimator (GET /v1/nodes/{name}/estimate) uses a node’s own results first, then those of the same device model. The hub does not send the optional on-demand{"type":"benchmark"}request yet. - Owner protections: kept per node from
helloand heartbeats. With the marketplace on, listings of nodes that reported none (older apps) are not offered to other owners (routing,/v1/route,/public/v1/market); their own owner keeps using them. - Recipes: after the welcome (and after every recipe upload or delete) the hub sends
recipes;recipeStatusis recorded per recipe version and node, and counted per device on the public registry. - Pending nodes: keys created by pairing start pending when
requireNodeApprovalis on; they connect but are never routed (PAIRING.md). - Revocation: revoking a node key closes its sessions with 4401 at once, including one still in its handshake, and live sessions are re-checked against the key store every 60 s.
- Direct connections:
directadvertisements are kept per node (admin-only inGET /v1/nodesand the dashboard), the welcome carries the ticket keys, anddirectUsageitems go into the audit log and usage withsource: "direct"(DIRECT.md). - Chat:
task: "chat"models are served on/openai/v1/chat/completions(nodes first, then any configured cloud provider), and, translated by the hub to the same chat run, on/openai/v1/responsesand/anthropic/v1/messages(the node sees an ordinary chat request); chunks are relayed to the client as they arrive (SERVER.md). - Test node:
server/cmd/fakenodespeaks this protocol with stub models (-models stub-llm:chatadds an echoing chat model that streams);server/scripts/e2e.shruns hub + fake node + requests end to end.