Routing
A client asks for a model by id (POST /v1/models/{id}/run, or model in an OpenAI request). The hub picks the node:
- Only online nodes that offer the model are candidates.
- Nodes where the model is already loaded come first.
- Then the lowest latency for that model: an exponentially weighted moving average of measured round trips.
- Then the fewest jobs in flight.
A node runs at most maxInFlight jobs at once (default 2). Further requests wait up to 10 s for a free slot, then get
429 busy.
Failover. Only transport failures are retried: if a connection drops with jobs in flight, they are retried once on another node with the model, else the client gets 503. A node’s own answer, even an error, is final, and so is a hub timeout (504).
Answers carry servedBy (the node) and via (the hub). Errors pass through unchanged, except that a node
answering 401/403 or garbage becomes 502 peer_unavailable.
Loops cannot happen: nodes never route, and a forwarded request carries X-Gateway-Hop: 1; a hop above 1 answers
508 loop_detected.
The live demo runs this same rule over the registry’s measured numbers.