Streaming

Overview

Streaming lets you receive partial output as it is generated, which improves perceived latency and user experience.

For OpenAI-style clients, prefer Responses streaming when the client expects the Responses shape. If the client uses Chat Completions streaming, AnyPath supports that compatibility path too.

Streaming is available on shared routes whose channel has passed streaming certification: the channel probe makes a real streaming request and verifies terminal SSE usage evidence (recorded as an sse_terminal row in channel_model_usage_evidence). A streaming request to an uncertified channel fails closed with model_unavailable; AnyPath does not guess usage or provider credit.

Where streaming works

• Shared capacity: streaming works on channels that have passed streaming certification. Streaming requests to uncertified channels fail closed with model_unavailable.
• Chat Completions: the client request must include stream_options.include_usage: true; the gateway does not inject it, and a streaming request without it is rejected. Cross-protocol streaming from a non-Anthropic client toward an Anthropic channel (translated Chat) has the same include_usage requirement.
• Anthropic Messages, Gemini :streamGenerateContent, and Responses: terminal usage is always present in the stream, with no extra client field required. Chat streaming over the Gemini protocol is not supported.
• Images: the images family does not support streaming.

How streaming settles

Streaming and non-streaming requests share one settlement loop: a balance hold is pre-authorized before upstream dispatch, the terminal SSE usage evidence determines metered usage, and the hold is captured. Capture never exceeds the hold.

If the client disconnects mid-stream, the gateway keeps reading the upstream stream to its terminal state — the provider still completes the generation — and settles from the terminal usage evidence for the actual generated usage. A disconnect is not a free request: the provider completed the work, so the buyer pays for the metered usage. The hold is released and the buyer is not charged only when the provider errors, the stream exceeds the 600-second timeout ceiling, or the terminal state genuinely lacks usage evidence. A streaming request is a single attempt with no fallback.

Choose the endpoint shape

Choose the streaming endpoint that matches the client’s API shape:

• Use POST /v1/responses when the client already uses Responses.
• Use POST /v1/chat/completions for OpenAI Chat Completions clients.
• Avoid building separate streaming code for each model.

What to watch for

• partial deltas should append incrementally
• long-running sessions should handle disconnects as normal failure modes
• retries should preserve the original API type whenever possible

Responses Streaming

curl https://api.anypath.dev/v1/responses \
  -H "Authorization: Bearer sk-your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "input": "Write a short poem.",
    "stream": true
  }'

from openai import OpenAI

client = OpenAI(
    api_key="sk-your-api-key",
    base_url="https://api.anypath.dev/v1"
)

stream = client.responses.create(
    model="gpt-4o-mini",
    input="Write a short poem.",
    stream=True
)

for event in stream:
    if event.type == "response.output_text.delta":
        print(event.delta, end="", flush=True)

import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: 'sk-your-api-key',
  baseURL: 'https://api.anypath.dev/v1'
});

const stream = await client.responses.create({
  model: 'gpt-4o-mini',
  input: 'Write a short poem.',
  stream: true
});

for await (const event of stream) {
  if (event.type === 'response.output_text.delta') {
    process.stdout.write(event.delta);
  }
}

Chat Completions Streaming

If your framework still expects SSE chunks from /v1/chat/completions, that also works. Set stream_options.include_usage: true so the stream carries terminal usage; the gateway rejects Chat Completions streams without it:

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Write a short poem"}],
    stream=True,
    stream_options={"include_usage": True}
)

for chunk in stream:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)

Stream End Conditions

Typical completion conditions:

• response.completed for Responses API streams
• finish_reason: "stop" for Chat Completions streams
• finish_reason: "length" when a token limit is hit
• tool/function call events when the model wants to use tools

Web App Pattern

async function streamChat(message) {
  const response = await fetch('https://api.anypath.dev/v1/chat/completions', {
    method: 'POST',
    headers: {
      'Authorization': 'Bearer sk-your-api-key',
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      model: 'gpt-4o-mini',
      messages: [{ role: 'user', content: message }],
      stream: true,
      stream_options: { include_usage: true }
    })
  });

  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    const chunk = decoder.decode(value);
    const lines = chunk.split('\\n').filter(line => line.startsWith('data: '));

    for (const line of lines) {
      const data = line.slice(6);
      if (data === '[DONE]') return;
      const parsed = JSON.parse(data);
      const content = parsed.choices?.[0]?.delta?.content;
      if (content) {
        document.getElementById('output').textContent += content;
      }
    }
  }
}

Best Practices

Prefer Responses streaming for new builds

Use /v1/responses if the client expects Responses streaming. Keep /v1/chat/completions streaming for clients that require the Chat Completions shape.

Flush output incrementally

Append delta chunks to the UI or terminal as they arrive rather than waiting for the full response.

Handle disconnects and retries

Treat network drops as normal failure modes and reconnect carefully for long-running sessions. If your client disconnects mid-stream, the gateway still finishes reading the upstream response and charges the request for the actual generated usage — disconnecting does not cancel the provider’s work or waive the charge.

Handle stream errors

A stream can fail mid-flight. When it does, the gateway truncates the stream: the connection closes without a terminal event — no error chunk and no data: [DONE]. This is deliberate: a truncated stream can never be mistaken for a complete response.

Treat any stream that ends without its protocol terminal event (response.completed for Responses, message_stop for Messages, a final usage chunk followed by [DONE] for Chat Completions) as failed. Record the X-Request-Id response header before retrying so support can trace the exact attempt.

Next steps

• Pair this guide with the API overview when you decide which API type to use
• Record request IDs when you debug latency or repeated work
• Read the error handling guide for retry and failure patterns