Model Serving

Model serving is the infrastructure and process that makes a trained AI model available for use in applications. It involves loading the model into memory, exposing it through an API endpoint, handling incoming requests, running inference (generating predictions), and returning results — all while managing performance, scaling, and reliability. Model serving frameworks (such as TensorFlow Serving, TorchServe, Triton Inference Server, and vLLM) provide the tooling to deploy models as services. They handle concerns like request batching (grouping multiple requests for efficient GPU utilisation), model versioning (deploying new models without downtime), health monitoring, and auto-scaling. For large language models, serving is particularly complex due to their size (requiring multiple GPUs), auto-regressive generation (producing tokens one at a time), and variable response lengths. Specialised LLM serving solutions optimise for these challenges with techniques like continuous batching, KV-cache management, and speculative decoding.

The gap between a working model in a notebook and a reliable production service is substantial. Model serving bridges this gap, ensuring that AI capabilities are available to applications and users with the performance, reliability, and scale required for business operations. Key business considerations include latency (how fast responses are returned), throughput (how many requests can be handled simultaneously), cost (compute resources required), and reliability (uptime and error handling). Different applications have different requirements — a real-time chatbot needs low latency, while a batch document processing system prioritises throughput. Organisations can serve models through cloud provider managed services (lowest operational overhead), self-managed infrastructure (maximum control), or hybrid approaches. The choice depends on cost sensitivity, performance requirements, data privacy constraints, and in-house expertise.