- what picks the provider
- the model's pattern, then the dialect the request arrived in
- a request in the provider's dialect
- passed through byte for byte
- a translation that drops a capability
on_fidelity_loss:rejectby default, orallowand count- the answer without the call
POST /v1/preflight
The request names an alias
A client asks for fast. The provider that serves it decides what fast means. The alias is advertised in GET /v1/models, so a client can use it without knowing what it resolves to today. That is the purpose of an alias, and it is also the gap. On a failover the candidate resolves the alias with its own table, which may name a different model from the one the primary would have sent. The request stays the same and the model that answers it changes.
Capabilities move the same way. An eval passes on one endpoint, with a particular set of things honoured: strict tools, structured output, prompt caching, the cache and reasoning token splits a budget is metered against. None of that is in the request either. A route that honours less still returns a 200.
A provider entry speaks one dialect
A provider entry is a name, a dialect and a URL. The dialect is openai or anthropic, and it says what the endpoint accepts. There is no third value.
Vendors host each other's dialects. Anthropic serves its Messages API and an OpenAI-compatible endpoint. Bedrock serves both, split by model family. Gemini serves only an OpenAI-compatible endpoint. Each surface is one provider entry in its own dialect, so a vendor with two surfaces is two entries claiming the same models. Two entries may claim one model only when their dialects differ. Two in the same dialect are refused at load, because file order would decide between them and nobody wrote that on purpose. Add a custom provider.
The request's dialect picks the entry
The route a request arrives on fixes its dialect. /v1/messages is an Anthropic-dialect client. /v1/chat/completions and /v1/responses are OpenAI-dialect clients. Among the entries claiming the model, the first that speaks the request's dialect is chosen, and the request is passed through. When none does, the first declared is chosen and the request is translated to it.
With both Anthropic entries declared, Claude Code on /v1/messages reaches the Messages API untouched, and an OpenAI SDK on /v1/chat/completions reaches Anthropic's OpenAI-compatible endpoint untouched. Anthropic does the translation in the second case, on its own infrastructure. Declare only the Messages API and Gate translates the OpenAI SDK's request instead, under the guard below. A passed-through body is never rebuilt, so a field Gate has never heard of reaches the provider intact. Fidelity.
What the gateway claims
Every fidelity claim is about code this process runs. A translator's losses are a table in the gateway, and a fixture per capability runs through the real translator on every build to hold the table to what the code does. A loss the guard names is one the release's own tests have seen.
The gateway makes no claim about another party's server. A vendor's compatibility endpoint is a provider entry in its own dialect, passed through like any other. What that endpoint keeps is in the vendor's documentation.
What a translated route drops
Two dialect pairs are wired. Each is the case no vendor fills: a client that speaks one dialect in front of a backend that speaks the other. Where a vendor hosts a compatibility endpoint, taking it is strictly better, because the translation then runs on their infrastructure and they own the drift.
| inbound dialect | upstream dialect | what it does not preserve |
|---|---|---|
openai | anthropic | audio_input, multi_choice, usage_details |
anthropic | openai | strict_tools, prompt_caching, structured_output, usage_details |
Each pair carries one route, the chat surface of its inbound dialect. /v1/responses, /v1/completions, /v1/embeddings and /v1/realtime exist only in the OpenAI dialect. One of them in front of an anthropic entry is refused with 422 route_not_translatable, before the budget hold and before anything is sent. Translation.
A lossy translation is refused or counted
Before a translated request is forwarded, Gate reads which capabilities the body uses. A request that uses one its translation would drop is decided by on_fidelity_loss, set for the deployment or per provider. reject is the default. It answers with a structured error naming every such capability, so the request is fixed in one pass. allow forwards anyway and counts the downgrade. The read is a bytes.Contains prefilter, then a structural confirm, memoized. It never runs on a same-dialect request, where there is nothing to decide.
usage_details is in both rows of the table, and no probe can read it, because it is a property of the response. Translated usage carries no cache or reasoning token splits, in either direction. A budget typed on a split is charged by total tokens on a translated path, and each such settlement is counted as metered_by_total. No translated request meters zero.
A translated request can also leave out a field the provider's dialect requires. max_tokens is optional in the OpenAI dialect and required by the Anthropic one. Gate refuses such a request with 400 required_field_missing, naming the field and both of its OpenAI spellings. It fills in no default, because any number it chose would cut long answers short with only finish_reason: "length" to say so. Configuration reference.
A profile narrows what can serve it
A key names a profile, and nothing on a profile is per-key, so the envelope is edited once for everyone holding it. A key that needs its own envelope gets its own profile, which keeps the answer in one place. A profile's keys take models and pool members out of the running before anything is dialled, and its access rules take the rest.
| on the profile | what it takes out of the running | counted as |
|---|---|---|
allowed_models | models outside the list, matched on the name asked for or on what an alias resolves it to | model_denied |
pool_subset | members outside one named subset of whichever pool serves it | pool_subset_missing |
access_rules | whatever the profile's own rules deny, after the deployment-wide ones | the rule's own code |
The right-hand column is the same envelope seen from a failover candidate. A candidate has to enter it too, so one the profile excludes is passed over untried, and kirto_gate_failover_skipped_total carries the reason it would have given. The request's own refusals travel the same way: a candidate that would answer route_not_translatable, fidelity_loss or required_field_missing is skipped too. A candidate that is always skipped is a failover policy that does not do what it reads.
Two teams on one deployment can hold different envelopes over the same providers: one pinned to the models it has evaluated, another free to take whatever answers. The providers are the deployment's. What may serve a given caller is the profile's. Give a team its own policy.
Failover admits each candidate like a primary
A provider's retries run out before its failover begins. Each attempt is admitted under the same key, budget and guardrail decisions as the first, and the candidates are walked from that provider's own failover.to list in order. A candidate's own list applies when that candidate is the primary, and never here.
The failures that move a request on are connect, 5xx, 429 and timeout. A 4xx other than 429 never fails over, because the same request would fail everywhere. timeout is opt-in and has to be named, because the abandoned request may still run to completion upstream, and bill. Naming it obliges this provider and every candidate to set first_byte_timeout: an attempt that times out at the first provider and then hangs at the second still waits, once at each. A stream whose first frame has reached the client, a WebSocket session and an MCP call stay where they are.
A suspended provider is never silently rerouted to its failover list. Suspending a provider for mishandling data is one decision. Sending the same prompts to a different third party is another, and it belongs in the configuration where it can be read. On a failover that does happen, the model and the backend move as a pair: the candidate that served, and that candidate's own resolution of the alias. Both are metered and both are what the reconcile audit reads, so the two ledgers agree about what one request was, and the model that answered is a fact you can look up. Suspend a key, provider or rule.
A pool is asked, and the answer is checked
A pool is a set of self-hosted model servers reached under one name. Which server takes a request changes its latency, because each one holds a different prefix cache and a different queue. Gate leaves that choice to a scheduler. Either an inference gateway already in the path makes it, and Gate sees a provider whose base_url is that gateway's, or Gate asks the pool's endpoint picker over ext_proc and dials the server it names. A pool is a pool because the document says pool:. The data plane reads no cluster API and holds no credential for one.
A remote scheduler naming a destination is telling the gateway where to send a request, so every endpoint the picker names is checked against the set the pool's name resolves to at that moment. An address outside it is refused, logged and counted, and the request is spread over the members in turn instead. The check holds however well regarded the scheduler is, because what it bounds is forwarding to an arbitrary destination. Inference pools.
Every change of route is counted
Retries and failovers are successes from the client's point of view. A provider that is steadily failing is invisible in the error rate and obvious here.
| counter | labels | what a value says |
|---|---|---|
kirto_gate_failover_skipped_total | provider, reason | A candidate could not be admitted and was passed over untried. Always skipped is a failover policy that does not do what it reads. |
kirto_gate_failover_total | from, to, reason | An attempt was abandoned for another provider. Steady here is a provider failing steadily, with failover hiding it from clients. |
kirto_gate_fidelity_downgrade_total | provider, cap, action | A translated request used a capability its translation loses: allowed forwarded it lossy, rejected refused it, metered_by_total charged a split-typed budget by total tokens. |
kirto_gate_retry_total | provider, reason | An attempt was re-sent to the same provider after a wait. Rising under load is the early sign of a retry storm. |
kirto_gate_pool_picks_total | provider, outcome | How a pooled request was sent. A rising fallback is a pool that declares a scheduler and has stopped being scheduled. |
The decision is also available before the call. POST /v1/preflight says which provider would serve a request, whether it would be translated, what the body uses and what the translation would lose, and sends nothing. POST /admin/v1/preflight asks the same resolver on a named key's behalf, so reproducing a tenant's refusal needs neither their credential nor a temporary edit to the configuration. Silent degradation.
Threat model
The asset is the behaviour you validated before you shipped, and the capabilities it rested on. Nobody attacks it. A candidate takes the request while the primary is down and resolves the alias to a different model. A translated path drops strict tools or collapses the token splits a budget meters. A pool member answers instead of the one the scheduler named. Each arrives as a successful response, which is what makes it hard to see. The control is passthrough whenever a provider speaks the request's dialect, a guard that refuses a lossy translation by default, a profile that bounds what may serve a caller, and a counter on every route that changed. The limit is a vendor's compatibility endpoint: Gate passes it the bytes and makes no claim about what it keeps.
The entry this section expands, beside the others: threats and limits.
- Passthrough is the default and the body is never rebuilt. Where a provider speaks the request's dialect the bytes are forwarded intact, so a field Gate has never heard of reaches the provider unchanged.
- A lossy translation is refused before anything is sent.
on_fidelity_lossdefaults toreject, and the error names every capability the translation would drop, so the request is fixed in one pass. - The loss table is held to the code. A fixture per capability runs through the real translator on every build, so a loss the guard names is one the release's own tests have seen.
- Ambiguity is refused at load. Two entries claiming one model in the same dialect would leave file order to decide, so the gateway refuses to start.
- A route that cannot be translated fails early. An OpenAI-only route in front of an
anthropicentry is refused before the budget hold and before anything is dialled, with the reason named in the error. - Every retry, failover, downgrade and pool pick is counted. What answered a request is on the record, which is the part an alias otherwise hides.
What routing does not decide for you
- It checks capabilities, and behaviour is outside it. The guard reads what a translation drops. Two providers that both keep a capability can still answer differently, and no routing decision can tell you that.
- No claim about a vendor's compatibility endpoint. It is a provider entry in its own dialect, passed the bytes like any other, and what it keeps is in the vendor's documentation.
- A profile cannot require a capability. Fidelity is decided per provider by
on_fidelity_loss, so a profile that should never be downgraded is expressed on the providers it can reach. - Two translated pairs, and no more. A dialect that is not carried is cut on the way in, so there is no translation to get wrong.
- The member set is as fresh as DNS. The picker watches the Kubernetes API and Gate watches DNS, so a pod the picker already knows about can be one the gateway has not resolved yet.
- No Gateway API implementation. No GatewayClass, no Gateway object and no
status.parentswritten back.
The posts behind this
- Proxies and the unified API. One API for every provider is a good trade until somebody asks what the provider actually received. A proxy that parses your request into its own shape and writes it back cannot answer, because what survives is whatever that shape knew about the week it was written.
- Response caching. A response cache in front of the model saves the whole call, including the part where something decided about it. Caching belongs where the model's own cache is, and a gateway earns the provider's discount by leaving the bytes alone.