Nvidia Proposes Coordinated AI Memory Design to Boost Inference Efficiency
Nvidia published an article on November 5 outlining an AI memory approach. [[fact:fact-601fwk6bw]] The article says growing models, context windows, and agent applications are increasing GPU memory pressure, while HBM supply is tightening and prices are rising. [[fact:fact-n7id22ay4]] Nvidia proposes co-designing GPUs, CPUs, networking, storage, and software to use memory more efficiently rather than simply increasing GPU memory capacity. [[fact:fact-f46f7wvzw]] The approach includes using NVFP4 to compress model weights and KV Cache, moving temporarily inactive context to lower-cost storage, and using multi-GPU distributed deployment to free HBM. [[fact:fact-w1jythqxt]] Nvidia disclosed that Vera Rubin NVL72 increases per-GPU HBM capacity by only about 3%. [[fact:fact-wc0gclhjj]] Nvidia also said Vera Rubin NVL72 delivers up to 30 times the inference throughput per unit cost of GB300 NVL72. [[fact:fact-2xxnbq4sg]]