How Gate works

Every request to Gate runs through the same set of checks before it reaches the upstream provider, and the same set of records after the upstream responds. This page walks through what happens, from your client’s POST to the response in your terminal.

At a glance

client → Gate → upstream provider → Gate → client

Gate sits inline between your application and the model provider. You send a normal Anthropic Messages (/v1/messages) or OpenAI Chat Completions (/v1/chat/completions) request to Gate’s base URL, and Gate forwards it on, adding routing, security, caching, cost accounting, and an audit trail on the way through.

Which upstream Gate talks to depends on your billing mode:

• Paying through Gate. Gate forwards through one of the provider accounts your org has enabled, using its credentials, and bills the request against your prepaid balance. Today paying through Gate bills two upstreams: Amazon Bedrock and OpenRouter.
• Use your own keys. You point Gate at your own upstream and supply your own credentials. Gate works with any upstream that speaks the Anthropic Messages or OpenAI Chat Completions API, including Anthropic, OpenAI, Google Vertex AI, Google AI Studio, OpenRouter, and any other OpenAI-compatible endpoint. You pay that provider directly. Gate is a policy, optimization, and logging layer in front of your own account.

Note: Google Vertex AI is fully routable but is your-own-keys-only today. Gate does not yet bill Vertex usage against a prepaid balance.

Between client and upstream, Gate:

1. Authenticates your key and decides which upstream serves the request.
2. Rate-limits by key and org, and enforces any usage caps you have configured.
3. Screens the prompt for injection, jailbreak, PII, PHI, and credentials.
4. Caches the request (when applicable) and adds compression and prompt-cache hints so you get the most value per token.
5. Forwards to the upstream, retrying transient errors.
6. Screens the response for sensitive data on the way back.
7. Records the request (token counts, cost, security verdict, upstream) in your request log and audit trail, then settles the bill.

Each step is a feature you control. Here is the user-visible knob for each.

1. Authentication and routing

Two things decide where a request goes:

• The headers on the request (see Authentication).
• The billing mode, which Gate infers from the request shape. There is no per-key flag to set.

Every request must carry a Gate API key (it starts with sk-gw-), most commonly as Authorization: Bearer $GATE_API_KEY. The key binds the request to your org so it can be rate-limited, billed, and audited.

Your own keys: you choose the upstream

Send the X-Gate-Upstream-Url header and Gate treats the request as your own keys. It forwards your bytes to that URL, with network protections that keep a supplied URL from being used to reach internal systems. It never reaches into your org’s provider accounts, and the prepaid balance check is skipped. In using your own keys you also pass your own upstream credential (for example Authorization: Bearer <your-upstream-key>) alongside your Gate key.

curl -sS -X POST "https://gateway.constellationgate.ai/v1/messages" \
 -H "X-Gate-Api-Key: $GATE_API_KEY" \
 -H "Authorization: Bearer $ANTHROPIC_API_KEY" \
 -H "X-Gate-Upstream-Url: https://api.anthropic.com" \
 -H "Content-Type: application/json" \
 -d '{"model":"claude-sonnet-4-5","max_tokens":256,"messages":[{"role":"user","content":"Hello"}]}'

To save typing the header on every call, point your SDK’s base URL at Gate and pass the upstream once via your client config. Most agent tools (Claude Code, Codex) inject it for you. See the manual setup guides for per-tool setup.

Paying through Gate: Gate chooses the provider account

Omit the upstream header and Gate routes the request as paying through Gate. It picks an enabled provider account on your org by this precedence:

1. An explicit provider hint, if you send one.
2. The preferred provider you have pinned for that model on the Models page.
3. The cheapest enabled provider that serves the model.

To pin a provider explicitly, send the X-Gate-Provider header, or set "provider": "<name>" in the request body. The value can be a provider type (for example bedrock) or the name of a specific account.

curl -sS -X POST "https://gateway.constellationgate.ai/v1/messages" \
 -H "Authorization: Bearer $GATE_API_KEY" \
 -H "X-Gate-Provider: bedrock" \
 -H "Content-Type: application/json" \
 -d '{"model":"claude-sonnet-4-5","max_tokens":256,"messages":[{"role":"user","content":"Hello"}]}'

Note: X-Gate-Provider applies to paying through Gate only. It selects among your enabled provider accounts. X-Gate-Upstream-Url applies to your own keys only. It sets the upstream you supply yourself. Sending the upstream header always puts the request in using your own keys, so the provider hint is ignored there.

When several providers serve the same model, Gate routes to the most cost-effective eligible one automatically. You can always override that with a hint or a pinned preference.

If the request cannot be served (an unknown model, no enabled provider, a provider that does not serve the requested model, or a missing required header), Gate returns a 400, 402, or 403 with a short JSON explanation, often including a “did you mean” suggestion. It never makes a partial upstream call.

2. Rate limiting and usage caps

• Per-key rate limit. A sliding window, configurable per key in the dashboard.
• Per-org rate limit. A single window shared across every key on the org, set in the org’s entitlements.
• Usage limits. Cost, token, or request ceilings over a time window. Optional, and attached per key or per org.

Exceeding a rate limit returns 429 with X-RateLimit-* headers (limit, remaining, reset, and the scope that tripped). Exceeding a usage limit returns 429 with the limit’s id, type, threshold, current usage, and time window. Both block the request and record the breach to your audit trail. See Limits and retention for the configurable values and the exact 429 body.

3. Security screening

Inbound prompts and outbound responses are scanned for:

• Prompt injection and jailbreaks. Detection works in stages: a fast pattern-based pre-filter, an ensemble of purpose-built classifiers, and a large-model adjudicator for the highest-confidence calls. Every request is scanned. There are no trust shortcuts that skip the scan.
• Sensitive data. PII, PHI, credentials, API keys, and similar.
• Spend ceiling. An optional per-key cost guardrail enforced as a security criterion.

Each criterion has its own configurable sensitivity (High, Medium, or Low) and action: log only, flag, or block. You choose how detections are handled under Guardrails. When part of the detection stack is briefly unavailable, it fails closed rather than letting a request slip through unscreened. See Security and the Prompt-injection defense page.

4. Caching and compression

• Response cache. An identical request returns a cached response when your org enables it. Hits cost nothing (no upstream call, no tokens) and the response carries X-Gate-Cache: HIT.
• Prompt-cache injection. For Anthropic-family requests (served directly or via Bedrock), Gate automatically inserts cache_control breakpoints so the upstream charges a fraction of the input rate for the cached prefix on later turns. If you already set markers yourself, Gate leaves them alone.
• Compression. Text-bearing fields are losslessly rewritten with safe transforms before forwarding, and tool schemas are never altered. The model sees the same meaning, the request carries fewer tokens.

Typical impact: meaningful savings on token spend with no client changes.

5. Forwarding

Gate forwards with retries on transient upstream errors (5xx and 429). Streaming responses (SSE, chunked JSON) are scanned in-flight, so you do not trade latency for a post-hoc check. Upstream 429s are relayed back to you verbatim, so your client’s own backoff logic stays in charge.

If a provider is unhealthy, Gate returns 503 early instead of leaving your client to time out, and skips that provider rather than hammering it. These early rejections are recorded as uncharged, so you are never billed for a request that never reached a provider.

6. Response screening

The same security checks that ran on the input run on the output. A response that leaks a credential, leaks PII, or matches an output policy can be blocked or redacted, depending on your configuration. Streaming responses are scanned incrementally as tokens arrive.

7. Recording and billing

Every request creates a row in your request log with:

• request id, key, org, and user (when present)
• model, upstream, and billing mode (either payment method)
• token counts (input, output, cached) and computed cost
• a cost-calculation status that explains how the cost was derived (for example calculated, cache_hit, byok, or free_request)
• security verdict (per criterion)
• latency and HTTP status

It also writes an immutable entry to your audit trail (see audit trail). You can review any request from the dashboard or via the Messages view.

How the charge is computed

For paying through Gate, Gate prices each request against the provider catalog and verifies accuracy. When a provider reports the cost of a request in its response, Gate uses that figure. Otherwise it prices the reported token usage against the catalog rate for the routed provider. All arithmetic runs in integer micro-USD (1 USD = 1,000,000 µUSD), so charges never drift through floating-point rounding across millions of small requests. Your balance and transaction ledger are kept to the micro-USD, and the dashboard’s cent figures are a rounded display of the same underlying amount.

Before forwarding a request paid through Gate, Gate places a short-lived reservation for a conservative upper bound of the request’s cost, including a safety margin. This keeps a burst of concurrent requests from collectively overdrawing a low balance. The reservation is only a hold. It is released when the request settles, and you are always charged the real cost, never the estimate. If your balance cannot cover the worst-case bound, Gate returns 402 (insufficient_balance) before contacting the upstream rather than serving a request it cannot bill. A request whose price cannot be determined up front is refused with 503 (pricing_unavailable) rather than served at an unknown cost.

Requests on your own keys record cost_usd = 0 with a byok status. You pay your provider directly, so there is no gateway charge to compute.

The dashboard’s cost-verification view independently re-checks what Gate billed against what each provider charged, so you can confirm accuracy.

Conversations

When a client sends a multi-turn conversation, Gate identifies the session automatically by hashing the conversation prefix. The dashboard groups requests by session under Conversations, so you see whole exchanges instead of isolated calls. There is nothing to wire up on the client side.

Where Gate runs

Gate is a hosted, cloud service. You integrate by pointing your client at Gate’s base URL. There is no customer self-hosted or on-premises build of Gate itself today. The “self-hosted” option you will see for your own keys refers to the upstream: you can point Gate at a model server you run yourself (for example a self-hosted, OpenAI-compatible endpoint) and Gate will proxy to it like any other upstream.

What this looks like in practice

• A clean call. client → security passes → cache miss → forward → upstream returns → response scanned → record written and settled. Only a few milliseconds of Gate overhead on top of upstream latency.
• A cache hit. client → security passes → cache hit → response served from Gate with X-Gate-Cache: HIT. No upstream call, no token cost.
• A blocked call. client → security verdict is block → Gate returns the configured error → no upstream call, no token cost. The block lands in your audit trail with the reason.
• An unaffordable call paid through Gate. client → balance cannot cover the worst case → Gate returns 402 before any upstream call → nothing is billed.

Next steps

• Authentication: keys, headers, and your own keys passthrough.
• Security: guardrails, sensitivity, and detection actions.
• Limits and retention: rate limits, usage caps, request size, and retention.