Jones Codes

All posts

Agentic Research

Projects

What It Takes to Serve an LLM to a Million People

50 min read

When I type into Claude Code, my laptop runs the agent and the model runs on a GPU somewhere far away. We have an agent loop that reads files, runs tests, edits files, and uses the CPU to complete this work. The model is a GPU fleet shared by thousands of other people at the same moment, and coding agents use about a thousand times the tokens of a chat, which is mostly input. Most people think the AI is one thing, but it’s really two systems talking over HTTP. I’m trying to think about which one is actually hard to scale.

If I wanted to host my own models, say a few of them, for a lot of users, what would it actually take? Is it just Kubernetes plus GPUs? My intuition would be pods behind a load balancer and autoscaling on CPU, but that may not be the case. I’m going to dive in.

First, how’s an LLM API different from a normal one? A normal API request is short and stateless, while an LLM request is long, heavy, and holds onto memory the whole time it runs. A typical REST call finishes in tens of milliseconds, while an LLM response streams for seconds. A max-effort reasoning model can take minutes before the first answer token, and every in-flight request holds a KV cache on the GPU. That’s about 330 KB per token for a 70-billion-parameter Llama model with its KV cache at FP16. A 128K-token context is about 43 GB, a bit over half of an 80 GB H100.

In a normal API any pod can take any request, but I’m thinking that’s not true in the case of an LLM API because of this cache mechanism.

KV cache stands for key-value cache and is the model’s working memory of everything it has already read in a conversation. It lives in GPU memory on one replica, which is one machine or a group of GPUs that the model is split across. If a subsequent turn in a conversation with an LLM is sent to a different pod, then that pod has none of the key-value cache, which causes a reread of the whole conversation from scratch, which we call a re-prefill. For a 100k-token Claude Code session, that’s the difference between a fast reply and a long pause, and a cache read price versus a full input price.

What’s Actually on the GPU

So what’s actually on the GPU? Generating each new token means reading the whole model out of GPU memory, so for a regular dense model, generation speed is set by memory bandwidth and not raw compute. (Mixture-of-experts models like DeepSeek only read the experts each token uses, but the idea’s the same.)

I’ve learned there are two phases:

  • Prefill, which reads your whole prompt at once and is limited by compute.
  • Decode, which produces one token at a time and is limited by memory bandwidth.

The rough ceiling for one user is bandwidth divided by model size. An H100, which moves about 3.35 TB/s, with a 70-billion-parameter model at FP8, gives about 48 tokens per second. An H200 has 141 GB at 4.8 TB/s, and a B200 has about 180 GB at 8 TB/s. These are expensive.

So that’s why VRAM is the whole hardware conversation. More VRAM means a bigger model fits, with room left over for everyone’s KV cache. Faster VRAM means more bytes per second come off the memory, and since every token needs a full read of the weights, more bytes per second is more tokens per second. So why does quantizing to FP8 or FP4 make the model faster and not just smaller?

Quantizing is like compressing the model, so there’s less to haul out of memory on every single token. Is a slightly less intelligent model at twice the speed a good deal? I guess it depends on your use case.

Before you can scale anything, you have to fit the model on one machine. Weights = parameters × bytes. A 70-billion-parameter model is 140 GB at FP16 (two or more H100s) or 70 GB at FP8 (one H200 with room left over).

DeepSeek V3/R1 is 671 billion parameters and won’t fit on 8 H100s, which is 640 GB. It needs 8 H200s. Wow, that is massive when you think about it relative to a normal personal computer, even a beefy one. GPT-OSS-120B fits on one 80 GB GPU.

Splitting a model across GPUs can run over PCIe (about 128 GB/s counting both directions), but it’s slow, because the GPUs swap results constantly. NVLink (900 GB/s to 1.8 TB/s, also counting both directions) is what makes it fast. That’s why the unit is an 8-GPU box or a 72-GPU NVL72 rack.

The more of the model you can keep inside one fast wired box, the faster it runs, while processing over the network between boxes is the slow path. It’s interesting to think about a laptop with one GPU versus a frontier model that runs on a rack that weighs more than a car. The infrastructure to run these things is quite crazy.

So now let’s think about one user versus a thousand and what it takes to make this work. One user wastes almost the entire GPU. The economics only work when many people share each pass through the weights.

This is a huge difference from my own laptop use. A single Claude Code session is a terrible GPU workload on its own. Serving a million people is cheaper per token than serving one, which makes sense because one user leaves most of the memory bandwidth unused.

The simplest way to say it: the weights are the expensive part to move, and batching moves them once for everyone.

The trade-off curve that nobody seems to talk about is that you can make the GPU produce more tokens in total or make each user’s tokens come faster, not both.

At first I didn’t get why total tokens matter. I can see why you’d want each user’s tokens to come faster. But why care how many tokens the GPU makes in total? Turns out it has nothing to do with answer quality. It sets the cost. A GPU-hour costs the same no matter what, so the more tokens the GPU produces in that hour, the cheaper each one is and the more people one GPU can serve.

Here’s a real example. A B200 serving MiniMax does about 9,600 tokens per second per GPU when each user gets 43 tokens per second. Push each user up to 102 tokens per second and the GPU total drops to about 2,100. Cost per million tokens goes from 5 cents to 23 cents (InferenceX, November 2026).

Every provider is picking a point on this curve. Fast modes and premium tiers are just a different point. What point would you pick for your own product is the question at large.

Which brings me back to batching. I had a few questions. How do thousands of independent prompts get batched together, run on a GPU at the same time, and come back out as the right answer for each person? Why batch at all? Is the time lost sending the prompt over the wire to the GPU, or is it caching, or the GPU starting up? The deep dive below walks through each one.

You don’t batch because requests are slow to arrive. You batch because every token costs a full read of the model. Might as well make that read count for everyone.

Between the API and the GPU

So what sits between the API and the GPU? It’s not a load balancer per se in front of a GPU. It’s a stack:

  • Gateway: auth and rate limits per org: requests, input tokens, and output tokens per minute.
  • Router: picks the replica that already has your prompt prefix cached (llm-d, the Kubernetes Gateway API Inference Extension, NVIDIA Dynamo). It routes on KV cache usage and queue depth, not round robin.
  • Engine: vLLM, SGLang, and TensorRT-LLM do batching and memory paging on the GPU.

Big shops split prefill and decode onto separate GPU pools. DistServe reports up to 7.4x more requests within the same latency targets.

flowchart TD
    C["Client<br/>agent on a laptop, or a cloud sandbox"]
    G["API gateway<br/>auth · per-org rate limits (RPM / ITPM / OTPM)"]
    R["Inference router<br/>picks the replica with your prefix cached<br/>routes on KV-cache use + queue depth"]
    S["Engine scheduler (vLLM / SGLang)<br/>continuous batching · KV-cache pages"]
    P["Prefill GPUs<br/>read the whole prompt at once<br/>compute-bound → sets TTFT"]
    D["Decode GPUs<br/>one token per step for the whole batch<br/>bandwidth-bound → sets tokens/sec"]
    X["GPU fault (Xid 74/79, ECC)<br/>KV cache lost → re-prefill elsewhere<br/>node tainted and drained"]

    C -->|"HTTPS request"| G
    G -->|"429 if your limit is hit"| C
    G --> R
    R -->|"529 if the whole fleet is overloaded"| C
    R --> S
    S --> P
    P -->|"KV cache handoff (NVLink / RDMA)"| D
    D -->|"tokens streamed back over SSE"| C
    D -.->|"mid-stream failure"| X
    X -.->|"retry, or fall back to another model/provider"| R

Separate prefill and decode pools are the big-provider layout (Dynamo, llm-d). On a single vLLM box, both phases run on the same GPUs.

In a normal API, routing is dumb on purpose. Here routing is performance. Why does stateful routing scare you as an engineer? Sticky routing is basically just having to route back to the same pod as before. Why is that scary? Turns out there are a few reasons.

Sticky routing makes each server matter again. Cloud ops folks have an old way of putting this: servers are either pets or cattle. Pets have names, and when one gets sick you nurse it back. Cattle are numbered, and when one gets sick you replace it. Stateless APIs turned servers into cattle. Sticky routing turns them partway back into pets. The conversation itself isn’t lost, since the client re-sends it every turn. But each server holds warm caches that are expensive to rebuild, and losing one means a pile of re-prefills.

So, Kubernetes? Yes, but almost none of the default autoscaling behavior applies. Interesting. Google’s GKE guidance says don’t scale on CPU. Scale on queue depth or batch size. GPU memory looks full all the time because vLLM pre-allocates it.

Cold start is a killer: about 11 minutes for an 8-billion-parameter model in one BentoML test (image pull, unpack, load weights), and about 25 times faster by streaming weights instead.

OpenAI has written about running Kubernetes at 7,500 nodes (research and training clusters, 2021). Separately, in December 2024 a telemetry rollout overloaded the control plane, broke DNS, and took everything down for about four hours. In a normal API, scaling adds a pod in seconds. Here it’s minutes, and GPUs you might not be able to get. Do you scale at all, or just overprovision?

In a normal API, autoscaling is how you save money. With GPUs, overprovisioning is how you buy reliability, and autoscaling is the backup plan.

If you want to watch this play out, I turned it into a short animated story on Kubedrills: The Model That Took Eleven Minutes to Wake Up. It follows a third replica from Pending to serving during a traffic spike, then replays the same morning with an always-on spare and with streamed weights.

Nobody serving at scale is running one model on one cluster. OpenRouter doesn’t own GPUs. It routes each request to one of many upstream providers based on price, throughput, time to first token, and 429 rate. It falls back on 5xx and rate-limit errors and keeps you on the same provider while your cache is warm.

Its rankings page shows the top model alone at about 56 trillion tokens in a week, and press reports put the total around 200 trillion a week this fall. Wow, that’s a ton.

Turns out picking a provider is harder than sorting by price. The same open model isn’t the same everywhere. Providers serve different quantizations, so one host might run FP8 and another FP4, and OpenRouter’s own docs warn that quantized models can do worse on some prompts. That’s the Anthropic misrouting story further down, at marketplace scale: same model name, servers configured differently, and a 200 every time. So OpenRouter’s default routing doesn’t rank on list price. It compares what a good response would actually cost at each provider, and it weighs throughput, time to first token, rate limits, reliability, and whether tool calls come back well formed.

It’s also the throughput-versus-speed curve from earlier, and you can see it on a model’s provider list. Every provider already picked its point. Specialty hardware hosts sit at fast and pricey, and the cheap hosts sit at slower streams. A router lets you pick your point per request instead of per vendor: sort by throughput when a person is waiting, sort by price when it’s a batch job.

Google reported about 3.2 quadrillion tokens a month at I/O ‘26 on its own TPUs. Anthropic serves Claude on AWS Trainium, NVIDIA GPUs, and Google TPUs. AWS Bedrock routes across regions with cross-region inference profiles, or you buy provisioned throughput.

Three hardware platforms for one model is a choice. What does that buy you, and what does it cost?

Running one model on three chip families is like shipping one app on iOS, Android, and web: more reach, three times the ways to break. Sounds like if you can handle the complexity, you get benefits, but those benefits come with a cost.

When the routing is wrong, the scariest failures aren’t errors. They’re wrong answers served with a 200. I first assumed that hitting the wrong pod, or the wrong cache, would give you a wrong answer. Turns out a cache miss mostly just costs speed, because the new pod re-reads your prompt. Wrong answers come from landing on servers that are configured differently, which is what happened to Anthropic.

In Anthropic’s 2025 postmortem, some Sonnet 4 requests were misrouted to servers configured for 1-million-token context. That peaked at 16% of requests in one hour. About 30% of Claude Code users hit it at least once. Sticky routing made it worse because follow-ups went back to the same bad server. A separate TPU compiler bug corrupted token selection.

Sticky routing is good for caching and bad for blast radius. How do you even detect that quality went down in monitoring?

A 500 error pages someone. A worse answer with a 200 pages nobody, so it can go unnoticed for weeks while things go downhill for everyone.

Latency and Failures

For LLMs you measure two latencies: how long until the first token, and how fast tokens keep coming. Those are the important ones: time to first token (TTFT) and time per output token (TPOT).

Low-effort frontier models take about 3 to 9 seconds to the first answer token. Max-effort reasoning takes minutes (Artificial Analysis, early November 2026). The tail comes from queuing behind long prompts, prefill stalling everyone’s decode, and preemption: when KV memory runs out, vLLM evicts a request and later reruns its entire prefill. Chunked prefill gets 2.6–5.6x more capacity within tail-latency limits.

In a normal API, p99 is a slow database query. Here it’s someone else’s 100k-token prompt landing in your batch. That’s a strange thing to explain in a planning meeting, so the callout below has a grocery-store version, plus what you can actually do about big prompts.

So how do you keep latency down?

Caching is the same lesson as every other system I’ve built. It ties back to what I wrote in Where LRU Caching Actually Matters.

GPUs fail constantly at scale, and an LLM request can’t just be retried like a GET. Meta saw an unexpected interruption about every 3 hours on 16K H100s over 54 days. About half traced to the GPUs or their HBM memory. The common hardware errors are:

  • Xid 48: ECC
  • Xid 74: NVLink
  • Xid 79: fell off the bus

Operators taint and drain the node, which means marking the machine so Kubernetes stops sending it new work, then moving whatever’s running somewhere else before it gets repaired. When a GPU dies, the request’s KV cache dies with it. It has to be re-prefilled somewhere else. Public APIs don’t promise to pick up a half-streamed answer where it stopped, so in practice the client starts over (more on why in the callout below).

In Anthropic’s codes:

  • 429: you hit your limit
  • 529: the whole service is overloaded
  • 500: retry with backoff

Midstream errors can arrive after a 200.

Retries with jitter. Retry at only one layer: 5 layers that each try 3 times is 3⁵ = 243x load. Fall back to another provider or model.

What does a good failure look like to a user? This stuff is hard to grok, so here it is in plainer terms.

In a normal API, a failed request is a blip. Here, it throws away work the GPU already did, so the whole design is about wasting as little of that work as possible and telling the user plainly what happened when some of it gets lost.

Agents in the Cloud

So, the other half: agents in the cloud. Run the agent in the cloud and now you need two fleets: computers for the agent and GPUs for the model.

  • Anthropic Managed Agents: a stateless “brain” harness and a disposable “hands” sandbox. Decoupling them cut p50 time to first token by about 60% and p95 by over 90%.
  • Devin: same split, brain in their cloud, a VM per session.
  • Codex cloud: the sandbox goes offline with secrets stripped.
  • Copilot agent runs on GitHub Actions runners.
  • Sandbox platforms (E2B, Modal, Vercel, Cloudflare, AWS AgentCore): microVMs or gVisor, starting in about 100 ms to 1 second, roughly 5 to 14 cents per vCPU-hour, sessions up to 8 to 24 hours.

The agent box is CPU-bound, stateful, and idle most of the time waiting on the model. The GPU fleet is shared and request-scoped. They scale on completely different signals.

Here’s how one Claude Code task touches both.

What It Costs

Self-hosting is only cheaper if your GPUs stay busy, and even then, maybe not. The formula: GPU dollars per hour divided by tokens per hour actually produced (SemiAnalysis’s definition).

Llama 3.3 70B on a rented H100: $2.90 per hour, about 1,500 tokens per second. That’s 54 cents per million tokens at 100% busy, $1.08 at 50%, $5.38 at 10%. ($2.90 is DeepInfra’s dedicated-GPU rate, Nov 2026. At Lambda’s $3.99 on-demand it’s $0.74, $1.48, and $7.41.)

The same model on a hosted API (DeepInfra) is about 12 cents per million (blending input and output 8:1, like the benchmark), four times cheaper than your perfectly busy GPU (about six times at on-demand rates).

API prices for the same capability drop about 10 times a year, so buying a GPU on a 4-year depreciation is a bet against the curve.

When does self-hosting actually make sense? Privacy, control, a fine-tuned model, steady 24/7 load. This is interesting when thinking about big companies who are thinking about self-hosting models to get off API prices. And Bedrock doesn’t get you out of API pricing. On-demand bills per token like any API, and provisioned throughput is reserved capacity by the hour, so it’s not self-hosting.

What if we have a company with, say, 1,000 engineers who work 9 to 5, Monday through Friday? It’s not really a 24/7 load. So I did the napkin math for that case.

Finally, cloud or a machine in a closet. Renting GPUs is flexible. Buying them only wins if you keep them busy. H100 on-demand runs from about $4 per hour (Lambda, RunPod) to $11 to $12 an hour (GCP, Azure) (Nov 2026 prices). B200 runs about $5 to $14 an hour. A DGX B200 costs about $400K and draws about 14 kW. For how long?

An NVL72 rack is about $3 to $4 million and 120 to 140 kW, liquid cooled. Rough break-even for owning versus renting on-demand: about 25% to 35% sustained utilization, before staff and spares. At home: Mac Studio, DGX Spark, or RTX 5090 with Ollama for one person.

Where does a small company land? Where would I land for my own applications? So I ran the numbers for three kinds of companies: an early-stage startup, a small company, and a large company.

For my side projects, and for most early-stage startups, it’s the left column.

That table has one problem for agents. It blends input and output 8:1 with no caching, which is a chat shape. Agent traffic runs about 100:1 with a cache hit rate around 92%, both from earlier in this post. Run Sonnet 5.5 through that shape and the price per million drops from about $2.89 to about $0.39, roughly 7x cheaper. Self-hosting moves the other way. Those throughput numbers come from 1k–8k-token prompts, and agent sessions sit at 50–100k tokens of context, so fewer users fit per GPU. So for agent workloads, the frontier column is a lot closer than the table makes it look. The large company’s owned GPUs come to about $0.16 per million ($485k a month over 3 trillion tokens). Against $0.39, that 18x gap shrinks to about 2.4x at most, before the long-context hit to self-host throughput.

To close, I’ll say that scaling LLMs isn’t Kubernetes plus GPUs. It’s memory management, stateful routing, and caching wrapped in a Kubernetes shell.

And hosting your own models brings all of that with it. You’ve got to run and tune the serving stack, build the prefix caching and cache-aware routing, handle cold starts and GPU failures, and keep people on call to maintain it. With an API, most of that comes out of the box. You turn on prompt caching, set your retries, and you’re serving. So when the prices come out about the same, the API is a pretty good deal, because that price already covers all the work you’d otherwise be doing yourself.

What should someone thinking about self-hosting do Monday? Start on an API. If you’re using open models, put a router in front so you’re not tied to one provider’s point on the speed and price curve. Measure your cache hit rate and utilization, and only self-host when the math says so. I’ve really enjoyed this learning session. Thinking about where I’ll go next, it seems like the routing is a big piece of this. I think I will continue to advance my knowledge around the details of the implementation and dive deeper into the things that make this work. I wanted to see Kubernetes as a wrapper around LLM inference in action, so I built a companion piece on my Kubedrills site: The Model That Took Eleven Minutes to Wake Up. Claude and I built it together, and its numbers come from the same sources as this post.

Graph View

Recent Posts