Skip to content

Streaming

POST /v1/chat/completions (API credit) or /token/v1/chat/completions (API token) with stream: true

CloudService streams model responses using server-sent events (SSE) — the same wire format as OpenAI. Pass "stream": true in the request body and consume the response as a stream.

For OpenAI-compatible clients, set the base URL to https://api.yourdomain.example/v1 for API credit or https://api.yourdomain.example/token/v1 for API tokens before using this endpoint.

Why stream

  • Lower time-to-first-token. Users see text as soon as the model produces it.
  • Better UX for long responses. Avoid a 30-second blank screen.
  • Cancelable. Disconnect mid-response to stop usage charges for tokens you won't use.

Request

Same as a non-streaming chat completion, with one extra flag:

stream.sh
curl -N "$CLOUDSERVICE_BASE_URL/chat/completions" \
  -H "Authorization: Bearer $CLOUDSERVICE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "YOUR_AUTHENTICATED_MODEL_ID",
    "stream": true,
    "messages": [{"role": "user", "content": "Write a haiku about streaming APIs."}]
  }'

The -N flag disables curl's output buffering so you see chunks arrive in real time.

Response

A series of SSE events. Each event is a JSON object on a single line prefixed with data:. The stream ends with data: [DONE].

events
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1719228000,"model":"fable-5","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1719228000,"model":"fable-5","choices":[{"index":0,"delta":{"content":"Tokens"},"finish_reason":null}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1719228000,"model":"fable-5","choices":[{"index":0,"delta":{"content":" fall"},"finish_reason":null}]}

data: [DONE]

With the OpenAI SDK

The SDK handles SSE for you. Just set stream=True:

stream.py
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["CLOUDSERVICE_API_KEY"],
    base_url=os.environ["CLOUDSERVICE_BASE_URL"],
)

stream = client.chat.completions.create(
    model="YOUR_AUTHENTICATED_MODEL_ID",
    stream=True,
    messages=[{"role": "user", "content": "Write a haiku about streaming APIs."}],
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
print()

Cancellation

Disconnect the request to cancel. The provider stops generating, partial tokens are billed, no usage record is written for the cancelled call. (Implemented via HTTP connection close — the gateway sees the TCP RST and forwards it upstream.)

Notes

  • Streams are not buffered by the gateway. Bytes from the provider go straight to your client.
  • The gateway sets Cache-Control: no-store on streaming responses.
  • Request timeouts on streaming are 5 minutes. Longer streams may need a server-side reconnect.

Working streaming clients: Python, JavaScript, and cURL. Failure codes returned mid-stream are listed in Error Codes.