TensorRT

TensorRT is a software development kit (SDK) from NVIDIA designed to optimise trained deep learning models for maximum inference performance on NVIDIA GPUs. It takes a trained model and applies a series of optimisations to produce an engine that runs significantly faster than the original model. Key optimisations include precision calibration (converting from FP32 to FP16 or INT8 while preserving accuracy), layer and tensor fusion (combining multiple operations into single GPU kernels), kernel auto-tuning (selecting the fastest GPU kernel for each operation), dynamic tensor memory management, and multi-stream execution. TensorRT supports models from all major frameworks through ONNX import or native framework integrations. It is particularly effective for transformer-based models, convolutional neural networks, and other architectures commonly used in production AI systems.

For organisations deploying AI on NVIDIA GPUs (the most common AI hardware), TensorRT provides the maximum possible inference performance. Speedups of 2-6x over unoptimised inference are typical, with some models achieving 10x or greater improvements. These performance gains translate directly to business value: lower latency for real-time applications, higher throughput for batch processing, and reduced compute costs for inference workloads. Given that inference costs often dominate the total cost of ownership for AI systems, TensorRT optimisation can significantly improve AI economics. TensorRT-LLM, a specialised variant for large language models, has become an important tool for organisations serving LLMs. It supports multi-GPU deployment, in-flight batching, KV-cache optimisation, and other LLM-specific techniques that maximise GPU utilisation and minimise response latency.