Most public conversation about AI infrastructure focuses on training: how many GPUs a lab used, how long a run took, how big the dataset was. But once a model is trained, it has to be served, and that turns out to be a distinct engineering problem with its own bottlenecks, tradeoffs, and cost structure. Understanding why inference is hard explains a lot about current hardware design, pricing models, and product decisions in the AI industry.

Training and Inference Optimize for Different Things

Training is a batch problem. You feed enormous amounts of data through a model over days or weeks, and you’re optimizing for total throughput: how much computation you can push through a cluster before the run finishes. Latency for any single example doesn’t matter. Utilization across thousands of chips does.

Inference is the opposite. A user sends a prompt and expects a response quickly, often in a chat interface where every additional second of delay is felt directly. Now you’re optimizing for latency per request, not just aggregate throughput, and you’re doing it for a huge number of small, unpredictable requests arriving at random times instead of one giant scheduled job.

Why Batching and Latency Pull Against Each Other

The standard trick for making any accelerator efficient is batching: grouping multiple requests together so the hardware processes them in parallel instead of sitting idle waiting for data. Bigger batches mean better GPU utilization and lower cost per request.

But batching also means requests have to wait for a batch to fill, or for other requests in the same batch to finish, which adds latency. A serving system has to constantly balance this: batch aggressively and save money but slow down individual users, or respond immediately and burn expensive compute cycles on underfilled batches. Techniques like continuous batching, where new requests can join an in-progress batch instead of waiting for the next one, exist specifically to soften this tradeoff.

Memory, Not Just Compute, Is the Bottleneck

A less obvious constraint is memory. Transformer-based models generate text one token at a time, and to do that efficiently they cache intermediate calculations, known as the key-value or KV cache, so they don’t repeat work for every prior token in the conversation. That cache grows with both the length of the conversation and the number of concurrent users being served.

This means inference capacity isn’t just limited by how much raw computation an accelerator can do, it’s limited by how much fast memory is available to hold all those caches at once. A chip with excellent compute throughput but limited memory bandwidth or capacity can still bottleneck badly under real serving load. This is a big part of why memory bandwidth, not just raw FLOPs, has become such a heavily marketed spec for AI accelerators.

Quantization: Trading Precision for Capacity

One of the main levers for fitting more inference workload onto the same hardware is quantization: representing model weights and activations with fewer bits than they were trained in. Lower precision means less memory used and often faster computation, at the cost of some accuracy.

The tradeoff isn’t always noticeable to end users, especially at moderate levels of quantization, but it isn’t free either. Different model architectures and tasks tolerate precision loss differently, which is why serving systems often support multiple quantization levels rather than a single default, letting operators pick a point on the cost-quality curve that fits their use case.

Why This Shapes the Business

Because inference cost scales with usage rather than being a one-time expense like training, it tends to dominate the long-run economics of any AI product with real traffic. That’s why so much industry effort now goes into inference-specific hardware, serving frameworks, and techniques like speculative decoding and model distillation. Training determines how capable a model is. Serving determines whether a product built on it can actually make money.

Understanding this split is useful even outside infrastructure teams. It explains why some AI features are slow to roll out broadly even after a model is technically ready, and why cost per query, not just model quality, ends up shaping what capabilities companies choose to ship.