Chapter 4 · Software
vLLM
3 of 5
4.3

Inference engines

Three engines are genuinely competitive: vLLM, SGLang, and TensorRT-LLM. All three give good out-of-the-box performance for LLMs and the architectures that resemble them, and all three support the techniques in Chapter 5: continuous batching, post-training quantization, speculative decoding, prefix caching, parallelism, disaggregation.

Their value is configurability. Working with pre-optimized components means your time goes into testing combinations rather than reimplementing the same attention kernel badly.

vLLM
UC Berkeley, now the PyTorch Project
Performance
Ease of use

The broadest support of the three, and by a wide margin the most popular. Nearly every open model works on day zero. Excellent when properly configured; usually short of what a narrower engine can extract.

SGLang
LMSYS Org
Performance
Ease of use

Comparable to vLLM in performance and ease, with particular strength in large MoE models and complex prompting via its flexible frontend. Runs on NVIDIA and AMD.

TensorRT-LLM
NVIDIA
Performance
Ease of use

The best performance available if you are on NVIDIA and willing to pay for it in configuration. Backed by NVIDIA's handwritten kernels, with a steeper learning curve, narrower model support, and no image or video generation.

No constraints, so all three are live. Reach for vLLM or SGLang to get running, and move to TensorRT-LLM when the last stretch of performance is worth the configuration effort.

Choosing an inference engine by what you actually need. All three support the same core techniques. Turn on the constraints that apply to you and watch the field narrow. The choice is usually made by requirements rather than benchmarks.

4.3.1vLLM#

vLLM has the largest share by a distance: more GitHub stars than the other two combined. Released in mid-2023 out of UC Berkeley, it now lives under the PyTorch Project at the Linux Foundation.

Breadth is the selling point: NVIDIA, AMD, and Intel GPUs plus Google TPUs, and day-zero support for essentially every open model. vLLM Omni extends it to image, audio, and video.

The tradeoff follows the principle from Chapter 1: constraints buy performance. A platform that must serve every model on every accelerator gives up some of the ceiling a narrower engine can reach. Configured well it is genuinely fast; it is usually not the fastest.

4.3.2SGLang#

SGLang, from LMSYS, is close to vLLM on both performance and ease. Its distinguishing angle is pairing a fast backend with a flexible frontend language, so you can customize one component without rewriting the rest. It works day zero with labs like DeepSeek, Qwen, and Kimi, and its investment in large-scale MoE serving is why it is the engine of choice at xAI.

4.3.3TensorRT-LLM#

NVIDIA’s engine, and the fastest of the three when you can use it. Its edge is access: NVIDIA engineers write its kernels, some of them closed-source, tuned hard for Hopper and Blackwell. The older 0.x releases built literal TensorRT engines (an artifact per model, precision, and GPU), but the modern version is a standalone PyTorch-based package served much like the other two.

The cost is everything else. The learning curve is steep, model support is narrower, new architectures arrive later, it is NVIDIA-only, and it does not do image or video generation.