Gateways and AI APIs

Every chat window, coding helper and RAG tool talks to its model over HTTP. Once you can read one chat request and its response, you can call any model yourself, and see what a gateway adds on top.

One request shape, many models

Most AI services and local engines accept the same request format, first made popular by OpenAI's chat completions API. Ollama, LM Studio and vLLM all offer it on your own machine, and DL Prism uses it as its main endpoint. Learn it once and you can talk to all of them by changing the address and the model name.

The request is a POST to a path ending in /chat/completions, with a JSON body. If HTTP methods, headers and bodies are new to you, read HTTP basics first.

The request

An illustrative request:

POST https://api.example.com/v1/chat/completions
Content-Type: application/json
Authorization: Bearer YOUR_API_KEY

{
  "model": "example-model",
  "messages": [
    { "role": "system", "content": "You are a concise assistant. Answer in one sentence." },
    { "role": "user", "content": "What is a context window?" }
  ],
  "temperature": 0.2,
  "max_tokens": 200,
  "stream": false
}

Field	What it is
model	Which model should answer. Each service has its own names
messages	The conversation so far, oldest first. Each message has a role and content
temperature	How adventurous the token choice is, from lesson 1. Optional
max_tokens	The longest answer you will accept. Optional
stream	true to receive the answer piece by piece. Optional; false by default

The three roles:

Role	Written by	Used for
system	You, the developer	Standing instructions: tone, rules, format
user	The person asking	The questions
assistant	The model, earlier	Its previous answers, sent back so it can see them

Remember that the model has no memory. To continue a conversation, you send the whole list again, including the assistant replies, with the new user message at the end. Long conversations therefore get slower and more expensive with every turn.

The response

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1767225600,
  "model": "example-model",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "A context window is the most text, in tokens, that a model can take in at once."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 31, "completion_tokens": 19, "total_tokens": 50 }
}

• The answer is at choices[0].message.content.
• finish_reason says why it stopped: stop means it finished naturally; length means it hit max_tokens and the answer is cut off.
• usage counts tokens in and out. Cloud services bill by these numbers, usually priced per million tokens, with output tokens often dearer than input tokens.

API keys

A cloud service needs to know who is calling, to bill them and enforce limits. You send an API key, a long secret string, usually as Authorization: Bearer <key>. Anyone who has your key can spend your money and see what your account can see.

• Keep keys in environment variables or a secrets manager, never written in code, config files you share, screenshots or chat messages.
• If a key leaks, revoke it in the provider's dashboard and make a new one. Deleting the message or the commit is not enough: it may already have been copied.
• Local engines on your own computer usually need no key at all.

DL Prism follows this pattern. In prism.yaml you name an environment variable with api_key_env: ANTHROPIC_API_KEY instead of pasting the key, and Prism reads it when it starts. See Keep API keys out of the file.

What a gateway adds

An AI gateway sits between your tools and the models. Every tool talks to one address; the gateway decides which model answers.

 editor ───┐                     ┌─► Ollama on this computer  (free, private)
 scripts ──┼─► DL Prism  ────────┼─► a cloud provider         (pay per token)
 CLI tool ─┘   127.0.0.1:8130    └─► another cloud provider   (backup)
             rules · failover · cost ledger

DL Prism runs on your machine at http://127.0.0.1:8130/v1. Any tool that lets you change its OpenAI base URL can use it. See Connect your tools.

Routing: which model answers

model in the request	What Prism does
A model name or alias, such as llama3	Manual: that model answers
auto	Automatic: your rules choose

Rules are checked from top to bottom; the first match wins, and routing.default answers if none match. Prism looks at the request with quick text checks (not with a model) and works out the kind of task, the programming language, whether there are images and the rough size. From the beginner guide:

routing:
  default: llama3
  rules:
    - if: {task: code}
      then: qwen-coder

prism explain dry-runs a decision without calling any model:

prism explain "Write a Python function that removes duplicates"

{
  "chain": [{ "local": true, "model": "qwen2.5-coder:7b", "provider": "ollama" }],
  "mode": "auto",
  "profile": { "task": "code", "language": "python", "vision": false, "est_tokens": 12 },
  "reason": "rule 1 matched (task=code)"
}

(Trimmed. The full output also has an input-cost estimate.) See Routing rules.

Failover: when a provider fails

A failover chain lists backup models. If the first one's provider is down, rate-limited (429) or returns a server error (5xx), Prism tries the next. Errors that are your fault, such as a bad request or a wrong key, are passed straight back, because another model would not fix them. When a provider fails three times in a row, Prism moves it to the end of every chain for 30 seconds.

routing:
  fallbacks:
    claude-sonnet: [gpt-5, deepseek, qwen-coder]

End important chains with a local model: it keeps answering when the internet does not. See Failover.

Budgets: a ceiling on spending

Prism multiplies the tokens each provider reports by the prices you enter, and records the cost. With budget.monthly_usd set, a rule with budget_exceeded: true can send everything to free local models once the month's money is spent. The budget only changes routing through that rule; it does not block a request that names a cloud model directly. See Costs and budgets.

Who answered, and why

Every answer from Prism carries headers that say what happened:

Header	Example
X-Prism-Mode	auto
X-Prism-Provider	ollama
X-Prism-Model	qwen2.5-coder:7b
X-Prism-Reason	rule 1 matched (task=code)

Streaming

A long answer can take many seconds. With "stream": true, the service sends the answer piece by piece as Server-Sent Events (SSE): a response that stays open and delivers text events one after another. In the OpenAI style each event is a data: line with a small JSON chunk, and the stream ends with data: [DONE]:

data: {"choices":[{"index":0,"delta":{"content":"A context"}}]}

data: {"choices":[{"index":0,"delta":{"content":" window is"}}]}

data: [DONE]

(Chunks trimmed to the important field.) Your program joins the delta.content pieces in order.

Two things change when you stream:

• Errors can arrive halfway. Prism cannot fail over once part of an answer has reached your tool.
• Token counts may be missing, so a gateway may have to estimate the cost. Prism marks such records as estimated.

Not every streaming API uses this exact format. LocalRAG streams its answers as SSE too, but with its own named events, token, citations and done. See The event stream.

Try it in ApiWorkbench

You can call a chat endpoint with a saved request and check it with tests. This uses DL Prism from its beginner guide, running with prism serve. With Ollama alone, use http://localhost:11434/v1/chat/completions and a model you have pulled, such as llama3.2, and skip the header test.

1. Open a new request tab, choose POST and type http://127.0.0.1:8130/v1/chat/completions.

2. On the Auth tab choose Bearer Token and type dummy in the Token field. Prism accepts any value while server.api_keys is empty.

3. On the Body tab select JSON and type:

   {
     "model": "auto",
     "messages": [{ "role": "user", "content": "Explain how DNS works in two sentences." }],
     "temperature": 0.2
   }

4. Select Send. The first answer may be slow while the model loads.

   Pretty shows a chat completion. The Headers tab of the response shows X-Prism-Model and X-Prism-Reason.

5. On the Tests tab add Status equals 200, JSON field equals choices[0].message.role with assistant, and Header contains X-Prism-Model with an empty value. Send again: the response's tab reads Tests (3/3).

6. Select Save…, name it Chat (auto) and create a collection called AI APIs.

To watch a streamed answer arrive, open a Server-Sent Events tab, choose POST, use the same URL, add the header Authorization: Bearer dummy on the Headers tab, use the same body with "stream": true added, and turn off Reconnect automatically on its Settings tab, so the app does not send the question again when the stream ends. See Server-Sent Events.

Stay safe

• Never put a real API key in a request you share. Store it as a variable of Type Secret in ApiWorkbench, which hides it in the console, docs and exports. It is hidden, not encrypted, so it still lives in the app's files on your device. See Secret variables.
• Watch spending. A loop or a script with a bug can send thousands of paid requests. Set a budget with your provider and, with Prism, a monthly_usd budget and a rule for it.
• Keep the gateway on 127.0.0.1. Prism listens there by default, so other computers cannot reach it. If you ever change server.listen to another address, Prism requires server.api_keys; set strong ones.
• Know where each request goes. With auto, a prompt may go to a cloud provider. Use prism explain, the X-Prism-* headers or offline mode (prism serve --offline) when a prompt must stay local.

Check yourself

QuizGateways and AI APIs

Question 1 of 6

In an OpenAI-style chat completion response, where is the model's answer?

Score: 0

What you learned

• A chat request is a POST with a model and a list of messages with roles; the answer is in choices[0].message.content.
• The model has no memory: you resend the conversation every time.
• API keys are secrets that spend money; keep them in environment variables and revoke them if leaked.
• A gateway gives every tool one endpoint and adds routing, failover and budgets.
• Streaming sends the answer as Server-Sent Events, piece by piece.

Next steps

• Tools and MCP
• DL Prism routing rules
• Tests without code in ApiWorkbench