Towards a Solution to the Management Scaling Paradox in Distributed LLM Inference

Disaggregated LLM inference separates prefill and decode across nodes, relying on cross-node KV cache reuse (prefix caching) to reduce time-to-first-token (TTFT). Yet existing systems manage this cache entirely in user space, atop the very kernel that already handles memory mapping, page tables, and RDMA registration. We identify a management scaling paradox: user-space overhead from coordination RPCs, per-chunk locking, redundant memory copies, and RDMA transfer fragmentation grows to 79% of achievable TTFT as the cache warms, nearly doubling latency when caching should help most. We formalize this paradox as a TTFT decomposition model and validate it on the state-of-the-art LMCache + Mooncake stack atop vLLM. We then present RMC (Remote Memory for Cache), an in-kernel framework that extends OS-provided shared memory (/dev/shm) to cluster-wide scope via content-addressed Virtual Memory Regions (VMRs), where any node computes the address of any cached chunk by hashing its token content, requiring no coordination. Across four workloads, RMC achieves 1.1–1.3× TTFT reduction over LMCache + Mooncake and up to 2.1× over full recompute.