Supermicro, IBM, and Nvidia Launch KV-Cache Platform for Enhanced AI Inference

On November 5, 2026, Super Micro Computer, IBM, and Nvidia announced a collaborative key-value (KV) cache platform designed to enhance large-scale AI inference. This innovative solution addresses the costly issue of GPUs performing redundant calculations by retaining context information already processed.

The KV cache platform aims to minimize the need for GPUs to recompute context when it is evicted from memory. By maintaining this internal record, the platform significantly accelerates response times for AI models, particularly in scenarios requiring long-context processing, such as retrieval-augmented generation (RAG) and multi-turn interactions.

Performance Improvements

Benchmarks from a validated reference architecture indicate remarkable performance enhancements. The new setup achieved time-to-first-token (TTFT) speed improvements of up to 56 times for contexts reaching 130,000 tokens. Additionally, the system saw a 22-fold increase in concurrent requests per second (RPS), climbing from 0.19 to 4.26, and a 95% reduction in total processing time for batches of 200 requests.

Even under competitive loads, the KV-cache platform maintained an 18-fold improvement, completing 200 requests in just over 55 seconds during noisy-neighbor tests.

Components of the KV-Cache Platform

• Supermicro’s Petascale Storage Servers: Provide the physical storage layer.
• IBM Storage Scale Software: Manages high-performance shared storage.
• Nvidia Dynamo: Handles inference software.
• Nvidia Spectrum-X Ethernet Switches: Facilitate data movement between components.

This platform is part of Nvidia's broader Inference Context Memory Storage Platform (ICMSP) initiative, which was first introduced at CES 2026. The collaboration between these three tech giants has been ongoing since mid-2025, focusing on optimizing AI inference processes.