Multi-turn Context Sharing

Arrival: Scheduler concatenates origin_input_ids + output_ids to search the tree.

Match: Identifies longest prefix and retrieves memory pointers.

Allocation: alloc() reserves new slots exclusively for the uncached suffix.

Forward Pass: The Attention kernel processes the new tokens, pointing back to the locked cache tokens.

Committal: Newly generated KV cache is converted to bigram keys, aligned to memory pages, and inserted as new nodes in the Radix Tree.

TTFT Reduction: Bypassing prefill for large shared prompts reduces TTFT from seconds to milliseconds.

Throughput: Dramatically increases generation capacity for agentic loops.

VRAM Contention: Heavy reliance on efficient Least Recently Used (LRU) / Least Frequently Used (LFU) eviction policies to prevent HBM overflow.

The backbone of high-efficiency multi-turn frameworks. SGLang uses it to dominate vLLM in prefix-heavy agentic benchmarks.

Protected vs Evictable States: Using strict reference counting (inc_lock_ref) to lock nodes actively in use, and automatically placing zero-reference nodes into a priority eviction heap.

In-Batch Prefix Caching: Pausing subsequent requests in a batch until the first request completes the prefix compute, guaranteeing the entire batch rides on a single cache generation.

SGLang: The absolute leader in prefix-sharing topologies via RadixAttention.

vLLM: Automatic Prefix Caching (APC) implemented as an opt-in feature.

Candidates designing systems for "AI Customer Service Agents" must explicitly identify prompt caching and Radix Trees. Failing to deduplicate system prompts across users reveals a lack of production intuition.