AIGateway API

For AI Agents: the Markdown version of this page is at https://ankole.agentbull.com/en-US/docs/ai-gateway/index.md. The documentation index is at https://ankole.agentbull.com/en-US/llms.txt.

AIGateway is the unified AI boundary of an Ankole deployment instance. External applications, enterprise systems, and SDKs call it directly over an OpenResponses-compatible API; internal agents call the same surface for their model turns. Every call resolves a model selector against the provider bindings the operator configured, and the upstream credentials never leave the control plane.

This page documents the real routes, request shapes, and the boundary between stateless and stateful calls. The source of truth is the control-plane router and the Ankole.AIGateway module; treat this page as the map, not the contract.

Where it fits

AIGateway sits between callers and providers. A caller — an agent’s model loop, a console operator, or an external integration — presents a bearer token and sends an OpenResponses-shaped request. AIGateway resolves the selector, prepares the request, fans it out to the bound provider, and returns either a single JSON body or a stream. LLM, embedding, rerank, web-search, and web-fetch capabilities all flow through the same boundary.

The decisive property: a caller never sees a provider credential. The control plane owns the credentials and the routing policy; the caller owns only its token and its selector.

Authentication

Every endpoint under /api/v1/ai-gateway runs through the :ai_gateway_api pipeline and the RequireAIGatewayAccessToken plug. A request must present a bearer token in the Authorization header, and the plug accepts exactly two kinds:

• an agent token — an AIGateway API key whose subject is an active agent Principal; the call is scoped to that agent’s model bindings and selectors, with subject_type = "agent";
• an admin token — an active human admin console token; the call is scoped to the operator’s view of providers, with subject_type = "admin_human".

A missing or unverifiable token returns 401 with code: "invalid_token". There is no anonymous path.

curl https://ankole.example.com/api/v1/ai-gateway/responses \
  -H "Authorization: Bearer $AIGATEWAY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"model":"main","input":"Summarize the open incidents."}'

The endpoints

All routes live under /api/v1/ai-gateway. The transport an endpoint uses is part of its contract, not a preference.

POST /responses is the center of the surface. It carries the model turn and is the only endpoint that branches on transport and on state.

Stateless responses over HTTP and SSE

A stateless call is one request, one response. The caller sends the full input, AIGateway resolves the selector, calls the provider, and returns the complete body. Set "stream": true and the same endpoint switches to Server-Sent Events: AIGateway opens an SSE stream, writes typed events as event: <type>\ndata: <json>\n\n, and ends with data: [DONE].

curl -N https://ankole.example.com/api/v1/ai-gateway/responses \
  -H "Authorization: Bearer $AIGATEWAY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"model":"main","input":"Draft a release note.","stream":true}'

Stateless HTTP and SSE share one hard rule: they reject the stateful fields previous_response_id, conversation, and store. If a request needs continuation across turns, it must use the WebSocket path. A request that sends a stateful field over HTTP or SSE gets 400 with code: "stateful_responses_require_websocket" and a message naming the offending field.

Stateful responses over WebSocket

Stateful responses live on GET /responses upgraded to WebSocket. The upgrade hands the connection to AIGatewayResponsesSocket with the subject’s identity, a 30-minute idle timeout, compression, and a 128 MiB frame ceiling. The client sends no frame while it waits for a response, so the idle timeout is also the longest time one response can take; it matches the upstream model first-byte and idle budget. Over this transport a call may set store: true and continue an existing conversation with previous_response_id or conversation.

This is where the durable lifecycle lives. A stored response gets an id of the form resp_{uuid}. Subsequent turns reference it with previous_response_id; a stored conversation is referenced by conversation. The control plane owns the continuation rules, the durable history, compaction, response projection, and recovery — none of that is the caller’s job.

Retrieve a stored response later, stateless and over plain HTTP:

curl https://ankole.example.com/api/v1/ai-gateway/responses/resp_4f3c... \
  -H "Authorization: Bearer $AIGATEWAY_TOKEN"

Compaction is the one tool that trades a long stored history for a shorter one without losing the thread. It has no endpoint of its own: send a request whose input carries a {"type": "compaction_trigger"} item, and AIGateway answers with one compaction output item. Every transport works — POST /responses returns the body, the same call with "stream": true returns the reply as SSE events, and the WebSocket returns the same events. Send only the trigger and it compacts the stored conversation named by previous_response_id or conversation, and the reply carries the checkpoint id to continue from. Send history alongside the trigger and it compacts what you sent.

Provider routing

AIGateway resolves a model selector to a real provider binding before any upstream call. An Agent has eight built-in profiles: primary, light, heavy, coding, vision_fallback, web_search, web_fetch, and image_generate. The first five select language models. The last three select separate capabilities. An Agent can also have custom language-model profiles. An admin uses explicit provider entries. GET /models lists what the current subject can resolve, with optional OpenRouter-style filters (q, context, min_price, max_price, sort, modality filters).

Each provider row owns a credential pool. Provider kind, base URL, headers, settings, and capability declarations are shared by all members. A model profile points to the row and never names a pool member. AIGateway selects a healthy member by the configured fill_first, round_robin, least_used, or random strategy. The Console translates these strategy names for the selected UI language, while the API and stored values stay unchanged. A stateful thread stays on the same member when possible.

An attributed 429, 5xx, or transport failure cools down only the credential that made the request. AIGateway selects another member, rebuilds the provider request, and performs a bounded retry with exponential backoff and jitter. The Rust kernel performs one transport attempt at a time. AIGateway does not switch to another provider when the pool is empty.

chatgpt_subscription is an ordinary provider kind. Its OAuth credentials stay in the control plane, where token refresh runs under a row lock. The Agent Computer and external callers never receive those tokens.

Resolution can fail in two ways the caller should handle:

• 422 unknown_model_selector — the selector is not bound for this subject.
• 422 model_binding_not_configured — the capability and name are bound but the provider binding is incomplete.

A capability the bound provider does not offer surfaces as 422 unsupported_capability. A provider the operator disabled surfaces as 422 provider_disabled. These are configuration problems, not transient ones; retrying without changing the configuration will not help.

Error shapes

Errors use an OpenAI-compatible envelope. The body is {"error": {"code", "message"}}, and the HTTP status matches the failure class. The classes worth planning for:

• 400 — the request body failed validation: missing model, missing input, an invalid limit or top_n, a stateful field over HTTP, a malformed compaction input. The code names the field.
• 401 — the bearer token is missing or unverifiable.
• 429 — the selected provider’s credential pool is exhausted. The error code is credential_pool_exhausted; retry_at is present when AIGateway knows the earliest recovery time.
• 404 — a stored response, conversation, agent, or file was not found for this subject.
• 422 — the request is well-formed but the control plane cannot serve it: unknown selector, unconfigured binding, unsupported capability, disabled provider.
• 502 / 504 — the upstream provider failed. 502 covers transport and invalid-response failures (upstream_transport_failed, invalid_upstream_response, ai_gateway_request_failed); 504 is upstream_timeout. A client 4xx from the provider is passed through at its own status.

When an upstream returns an error.message, AIGateway forwards that message; otherwise it reports the upstream HTTP status plainly.

Image generation

image_generation is a public Responses tool with two execution paths. When the subject has an image_generate profile, AIGateway runs the tool with that separate provider and model. Without the profile, AIGateway passes the tool to the main provider only when its capability declares native image generation. If neither path exists, request preparation fails instead of simulating the tool.

Both paths use the same public stream events and generated-image persistence. Model usage and image usage stay attributed to the credential that produced each part.

Web tools, files, and the other capabilities

The same subject and token drive the adjacent capabilities. POST /web_search takes a length-bounded query and returns provider-backed results; POST /web_fetch takes one to five public HTTPS URLs and returns page content. These calls can use web_search.default and web_fetch.default, which resolve the current Agent profiles.

POST /embeddings accepts text, token arrays, or input blocks. POST /rerank reranks a non-empty document array and takes a positive integer top_n. These two endpoints require an explicit provider_id/model selector; they do not resolve Agent profiles. Brain uses the instance-wide brain.embedding_model and brain.rerank_model settings from AppConfigure when it calls these capabilities. See Brain for the retrieval behavior.

Files are first-class: POST /files uploads, GET /files lists, GET /files/:id and GET /files/:id/content read metadata and bytes, and DELETE /files/:id removes one. All are scoped to the subject.

What AIGateway is not

It is not a public, unauthenticated proxy. It is not a place to send provider credentials — those live in the control plane. And it is not a queue or a job runner; long-running agent work belongs to the Actor Runtime and Background Agent Jobs. AIGateway is the request/response boundary: one call in, one response or one stream out, with selectors resolved and credentials kept inside.

Next steps

• For where AIGateway sits in the whole system, read the architecture overview.
• For running the server that hosts these routes, read the deployment section of Quick start.
• For the first Provider and model-profile setup, read Quick start.