More HBM Won't Fix Your AI Factory

Two NVIDIA engineering posts this week make an argument capacity planners should internalize: agentic inference has turned GPU memory into the scarcest resource in the AI factory, and the fix is not shoving more HBM onto each package. It is treating memory as a hierarchy and codesigning every layer — silicon, CPU, storage and software — to do more useful work per byte.

Agentic workloads eat HBM

During inference, HBM holds both model weights and the KV cache, the working memory that lets a model serve each active request without reprocessing context. Agentic applications stretch that working set hard: trajectories carry prior context across repeated model and tool calls, in some cases processing up to 3 million tokens, and retrieval-augmented generation piles enterprise data on top. The result is three-way contention for the fastest, most expensive memory tier in the building.

Capacity is not throughput

The obvious response — add HBM — runs into a limit NVIDIA frames with a warehouse analogy: capacity is warehouse space, bandwidth is the loading docks. Once the docks are saturated, extra shelves go unused while still adding cost and power draw. Past that point — the memory wall — added HBM cannot serve more concurrent users within a response-time SLA.

The numbers back the framing, per NVIDIA's engineering post S1. From Blackwell Ultra to the Vera Rubin platform, HBM capacity per GPU rose roughly 3%, yet agentic throughput per dollar rose by up to 30x on the SemiAnalysis AgentX workload; the prior GB300 NVL72 step was up to 10x per-dollar versus H200 at 100 tokens per second per user. The gains came from engineering, not from capacity.

What full-stack codesign actually changes

NVIDIA's post splits the problem into three moves. Shrink weights: NVFP4 checkpoints cut footprint versus FP8, and NVL72 systems spread a model across up to 72 GPUs. Offload idle context: Dynamo inference software moves inactive KV cache to CMX context memory storage shared across servers, while the Vera CPU's large LPDDR pool — connected to Rubin GPUs over NVLink-C2C — gives paused context a nearby place to wait. Keep RAG data out of HBM entirely: Nemotron Retriever embeddings and the cuVS index run on CPU and storage, so only retrieved passages enter context.

Takeaways for capacity planners

Measure what matters: on NVIDIA's telling, the useful metric is tokens per second per watt and per dollar within an SLA, not gigabytes per GPU. The CPU tier is part of the memory design: in NVIDIA's 100-megawatt data center comparison, Vera delivered 1.6x more agents per watt than the fastest x86 CPU and 2x faster agent speed than the most energy-efficient x86 part, and Daytona brought 1,000 agent sandboxes online in 10.8 seconds versus 88.4 on x86 S2. Faster tool execution also shortens how long KV cache sits idle before offload.

Finally, assume memory stays expensive. NVIDIA's own post notes supply is tightening and prices are rising, and the chip supply chain remains politically tangled: New York Times reporting relayed by The Verge found approved Chinese buyers of Nvidia's H200 chips including firms like New Matrix that operate out of an apparently empty office, with ties to an entity blacklisted since 2024 S3. Whether or not those sales proceed cleanly, the safe planning assumption is the same: squeeze more useful work out of every installed byte rather than betting on cheap, abundant HBM.