Larger models, longer contexts, and agentic applications are increasing pressure on scarce GPU high-bandwidth memory. Its response is full-stack codesign: designing GPUs, CPUs, networking, storage, and software together so AI factories can use a hierarchy that balances speed, capacity, power, and cost instead of relying only on more HBM.
The company identifies model weights and KV cache as major HBM consumers. NVFP4 can reduce the footprint of weights and cache compared with FP8, while NVL72 systems distribute model weights across as many as 72 GPUs. More of each GPU’s HBM can then remain available for active context and concurrent users.
For inactive context, NVIDIA’s Vera CPU provides nearby LPDDR memory, while Dynamo inference software can move KV cache to CMX context memory storage and return it before reuse. Dynamo’s KV Cache Reuse manages cache across GPU memory, CPU memory, and storage. Separately, Nemotron Retriever models create compact embeddings, and cuVS keeps retrieval indexes outside HBM so only selected passages enter the model context.
NVIDIA argues that useful throughput matters more than raw memory capacity. On a SemiAnalysis AgentX workload cited by the company, GB300 NVL72 delivered up to 10 times the throughput per dollar of H200, while Vera Rubin NVL72 reached up to 30 times the throughput per dollar of GB300 at a higher per-user token rate, despite HBM per GPU rising about 3%.