Skip to content

OpenAI-compatible API

Base URL: https://<hub-host>/openai/v1, API key: a virtual key (sk-aigw-…). Existing OpenAI clients work by changing base_url.

Method Path Notes
GET /models, /models/{id} {"object":"list","data":[{"id","object":"model","created","owned_by","task"}]}: node models plus configured chat aliases
POST /embeddings see below
POST /chat/completions node chat models (llama.cpp on kinherd-node) first, streaming and tool calls included; otherwise 404 unless openai.providers is configured, then passed through to the provider behind the alias
POST /responses the Responses API on node chat models, stateless (see below)

input is a string or an array of up to 64 items. Each item is a data:image/…;base64,… string, {"type":"image_url","image_url":{"url":"data:…"}} (or "image_url":"data:…"), or {"type":"text","text":…}.

  • encoding_format: float or base64 (little-endian float32, the SDKs’ default).
  • Remote http(s) image URLs are refused: the hub never fetches on your behalf.
  • Text goes only to text-embedding models; text sent to an image model is a 400 that says so.
  • dimensions, if given, must equal the model’s size. usage is 0.
from openai import OpenAI
import base64
hub = OpenAI(base_url="https://<hub-host>/openai/v1", api_key="sk-aigw-…")
def data_url(path: str) -> str:
return "data:image/jpeg;base64," + base64.b64encode(open(path, "rb").read()).decode()
res = hub.embeddings.create(model="clip-l14-336", input=[data_url("a.jpg"), data_url("b.jpg")])
a, b = (d.embedding for d in res.data)
print(sum(x * y for x, y in zip(a, b))) # cosine similarity (vectors are L2-normalised)

The hub is not an LLM runtime today. With openai.providers and openai.models aliases configured, it passes /chat/completions through to an upstream OpenAI-compatible API, with the same keys, scopes (on the alias name), rate limits, quotas and audit as local models:

openai:
providers:
- {name: openai, baseURL: https://api.openai.com/v1, apiKeyEnv: OPENAI_API_KEY}
models:
gpt-small: {provider: openai, model: gpt-4o-mini}

Errors on /openai/v1 use the same codes as the native API, plus OpenAI’s "type" field.

POST /openai/v1/responses runs on the same node chat models as chat completions, with the same keys, scopes, local-only rule, quotas and audit. It is stateless: nothing is stored, so send the whole conversation in input each time (previous_response_id, conversation and background are refused with 400; store is always answered as false).

  • input: a string, or message, function_call and function_call_output items; instructions becomes the system message.
  • tools of type function only; built-in tools (web search, file search, computer use) and file inputs are refused.
  • max_output_tokens is capped by the hub’s openai.maxTokens.
  • stream: true sends the usual events (response.created, response.output_text.delta, response.function_call_arguments.delta, …, response.completed).
  • Cloud provider aliases are not served here; use /chat/completions for them.
from openai import OpenAI
hub = OpenAI(base_url="https://<hub-host>/openai/v1", api_key="sk-aigw-…")
r = hub.responses.create(model="qwen2.5-1.5b-instruct", input="Name three uses of a spare phone.")
print(r.output_text)

tools, tool_choice and parallel_tool_calls pass through to llama.cpp (at most 128 tools, 256 KiB), and answers keep tool_calls with finish_reason: "tool_calls". The model must be set up for tools on its node ("tools": true, which starts llama-server with --jinja). Your client runs the tools; the hub and the node never execute anything.

Sealed (end-to-end encrypted) bodies are accepted only on the native run API, not on /openai/v1: the hub would have to read a request to translate it.