Chapter 0 · Inference
The three layers
2 of 6
0.2

The three layers

Doing this well means solving three separate problems, and they do not substitute for one another. A team can be excellent at one and still have a service nobody wants to depend on.

L3Tooling

The surface engineers work through

Owns
  • How a model gets deployed and rolled back
  • What is configurable and what is decided for you
  • Logs, traces, and metrics reaching the people on call
Built from
Deployment APIsConfig schemasSDKsDashboards
Skip it and

Every deployment becomes a bespoke ticket for the one person who understands the cluster. Nothing is reproducible, and the team's throughput is capped by that person's calendar.

L2Infrastructure

Across instances, regions, and clouds

Owns
  • Adding and removing replicas as traffic moves
  • Finding GPU capacity wherever it exists
  • Routing requests to the replica that can answer fastest
Built from
KubernetesAutoscalersLoad balancersCapacity pools
Skip it and

The service is fast until it is popular. Traffic doubles, replicas cannot be found in time, requests queue, and a perfectly optimized runtime times out anyway.

L1Runtime

One model, one instance

Owns
  • How many requests share a forward pass
  • Which kernels run and in what precision
  • What gets cached between requests
Built from
CUDAPyTorchvLLMSGLangTensorRT-LLM
Skip it and

You pay for more GPUs than the traffic needs, and no alert catches it, because nothing is failing: on every dashboard you have, overspend looks identical to health.

Figure 0.1. The three layers of an inference stack. Each layer owns a different problem, and a team can be excellent at one while another quietly fails. The layers are ordered by distance from the hardware: the runtime sits closest to the GPU, the tooling closest to the engineer.

The runtime is the narrowest of the three and the easiest to reason about: one model, one instance, and the question of how much useful work you extract from the GPU you are already paying for. It rests on a deep software stack that runs from CUDA up through PyTorch to inference engines like vLLM, SGLang, and TensorRT-LLM. Low-level work matters here in a way that surprises people coming from web services; a better attention kernel is worth more than most application-level cleverness.

The infrastructure layer exists because no single instance is enough. Whatever you do to one server, it will eventually receive more traffic than it can serve, and that is not a CUDA or PyTorch problem. It is a distributed-systems problem: replication, routing, and scaling.

The tooling layer is the one teams skip, because it feels like overhead next to the other two. It is the interface everything else is operated through, and its absence shows up as a team that cannot deploy without a specific person being awake.