Hardware

A spec sheet has three numbers on it that matter, and which one you should care about depends entirely on what you are running. Reading them correctly is most of what hardware selection is.

Inference runs on accelerators, and in the datacenter that overwhelmingly means NVIDIA GPUs. There are three broad classes: datacenter GPUs racked in standardized chassis, workstation GPUs for professional desktops, and consumer cards. Serving at scale means the first category, not because the silicon is categorically better but because those cards ship with high-bandwidth interconnects, arrive in predictable configurations, and exist by the million.

You are almost certainly renting them. Cloud is the default: hyperscalers like AWS and GCP, or neoclouds like CoreWeave and Nebius. On-premise and air-gapped deployments are real but belong to large enterprises and governments; everyone else wants the ability to add capacity without a procurement cycle.