Streaming

Streaming in the context of AI refers to the real-time delivery of model-generated text as it is produced, rather than waiting for the entire response to be completed before sending it to the client. This is typically implemented using Server-Sent Events (SSE) or WebSocket connections. Without streaming, a user submitting a query to a language model must wait until the entire response is generated before seeing any output. For long responses, this can mean several seconds of blank waiting. With streaming, each token appears on screen as soon as it is generated, creating a fluid, typewriter-like experience. Streaming also enables important engineering capabilities. Applications can begin processing partial responses immediately, implement early stopping if the response is going off-track, display progress indicators based on actual generation, and reduce perceived latency even when the total generation time is unchanged.

Streaming dramatically improves the perceived responsiveness of AI applications. Studies consistently show that users prefer interfaces that display information progressively, even if the total wait time is the same. For customer-facing AI products, this can significantly impact user satisfaction and engagement. From a technical perspective, streaming enables more sophisticated application architectures. Developers can parse structured responses as they arrive, trigger downstream actions before generation completes, and implement token-level monitoring for content safety. This is particularly valuable for applications that use tool calling or structured output formats. Streaming also provides operational benefits. It allows for time-to-first-token (TTFT) monitoring, which is a key performance metric for AI services. Teams can set latency targets, detect performance degradation early, and optimise their infrastructure based on real generation timing data.