How the hub and the nodes fit together
Apps know one address and one key. The hub knows your devices, what each chip runs well, and who called what. The nodes do the work and never open a port.
Hub
One static Go binary in Docker (about 17 MB, non-root, no shell). It runs no models: it checks keys, meters, audits and routes.
- OpenAI-compatible API
- /openai/v1/models, /embeddings (image data URLs; text for text-embedding models), /chat/completions with streaming and the Responses API, with tool calls, run by llama.cpp on your own nodes. The Anthropic Messages API is there too, so Messages API clients can point at the hub. A cloud provider is an optional fallback the operator can configure.
- Native API
- POST raw image bytes to /v1/models/{id}/run and get labels or a vector back, with servedBy, via and a timing breakdown.
- End-to-end sealing
- A client can seal a job to the node app that runs it, so the hub, the proxy and Cloudflare relay only ciphertext. The hub still sees who, when, which model and sizes.
- Virtual keys
- sk-aigw-… keys per app or node: model-glob scopes, per-minute rate limits, daily quotas, instant revocation. Only HMAC-SHA256 hashes are stored.
- Audit and usage
- One audit entry per request (key, model, node, status, ms, bytes; never bodies or keys), usage per key per day, Prometheus metrics on a private port.
- Router
- Among online nodes that have the model: loaded first, then the lowest measured latency, then the fewest requests in flight. A transport failure retries once on another node.
- Dashboard
- Server-rendered, no JavaScript, strict CSP: nodes, models, registry, keys, usage, audit and the waitlist.
Nodes
A node is a device running the node app or kinherd-node with a node key. It dials out to the hub over one TLS WebSocket, says what it has, takes jobs and benchmarks itself. It works behind NAT and on mobile data.
| Device | Runtime | Measured or how it installs |
|---|---|---|
| Android phones | LiteRT 2 on the GPU, NPU or CPU. | CLIP ViT-L/14-336 at about 1.45 s per image on a Pixel 7 Pro GPU. |
| Android TVs and boxes | LiteRT 1.4 with NNAPI on the NPU; 32-bit builds included. | EfficientNet-Lite4 int8 at 22 ms on a MediaTek MDLA (TCL Google TV). |
| Linux (amd64, arm64) | kinherd-node: TensorFlow Lite for image models, llama.cpp’s llama-server for GGUF chat models. | Installed with one command, as your own user. |
| Mac | The same kinherd-node, built from source, as a launchd agent. | Chat models through llama.cpp on Apple Silicon. |
No iOS node. Chat models run on Linux and Mac nodes, not on phones yet.
- No inbound port: with
"listen": falsea node has no network attack surface at all. - Per-model queue, memory budget, lazy load and idle unload; a full queue answers 429 instead of piling up.
- Reconnects with jittered backoff, pauses with no network and resumes when Wi-Fi or mobile data returns.
- On the phone, same-device apps can reach the models over Binder IPC, with no network at all.
Registry
A model × device table: works, works with tweaks (and the tweak), or fails (and why), with latency. Nodes fill it in automatically, a person curates it, and the router uses it.
TCL Smart TV Pro · MediaTek MT9689 · nnapi:mtk-mdla_shim
efficientnet-lite4-int8: works with tweaks, 22 ms
Load the model from a memory-mapped buffer, not a path on shared storage: SELinux blocks the NN HAL from reading FUSE files, and NNAPI then silently falls back to its CPU reference path (~9 s per image).

API
Use the OpenAI SDK you already have, or the native API when you want raw bytes in and timings out.
# raw image in, labels out; the hub picks the node
curl -H "Authorization: Bearer $KEY" --data-binary @photo.jpg \
https://<your-hub>/v1/models/efficientnet-lite4-int8/run
{"model":{"id":"efficientnet-lite4-int8","version":"1.0.0"},
"output":{"labels":[{"label":"…","score":…}, …]},
"ms":72,"backend":"nnapi:mtk-mdla_shim","servedBy":"living-room-tv","via":"hub",
"timings":{"decode":…,"render":…,"pack":…,"invoke":22.0,"postprocess":…}}Marketplace (planned, not available yet)
The idea: model makers publish a specialised model (a card recogniser, a pest classifier, a receipt reader) that installs onto the buyer’s own hub, with updates handled. It is off on every hub, and paused while its privacy design is worked out.
What exists today is the package format underneath it: a manifest (inputs, outputs, accelerator preference, memory, licence, SHA-256) plus the model file, signed with Ed25519 by a publisher key that the hub and the node trust. Unknown fields and unsigned packages are refused.
Marketplace not open yet: not available in the beta
Where the hub runs
On your own machine: a server, a Pi, a NAS, a mini PC. Requests and data stay on your network. That is the only way to run it today.
There is no hosted hub and no cloud control plane. Both have been thought about (multi-tenancy, EU data residency); neither is built, and there is no date.