Inference Autoscaling Systems
Modern inference autoscaling systems explicitly disaggregate the LLM generation process into entirely separate prefill and decode hardware pools.
Source: mortalapps.com- Modern inference autoscaling systems explicitly disaggregate the LLM generation process into entirely separate prefill and decode hardware pools.
- Their core purpose is to meet strict latency Service Level Objectives (SLOs) independently for TTFT (Time to First Token) and TPOT (Time Per Output Token).
- The primary optimization idea relies on architectures like "DistServe" and "Splitwise," decoupling the compute-bound phase from the memory-bound phase.
- The most important engineering insight is that co-locating prefill and decode phases causes severe interference; autoscaling them separately maximizes per-GPU "goodput."
Why This Matters
In standard monolithic serving architectures, a sudden influx of requests with massive prompts (prefill) monopolizes the GPU's compute cores, catastrophically stalling the generation of tokens for existing requests (decode). To meet latency SLAs under these conditions, operators are forced to severely underutilize their GPUs by enforcing low batch sizes, or aggressively overprovision hardware. Disaggregated serving solves this fundamental conflict, with research demonstrating that such systems can serve up to 
more requests or achieve 
tighter SLO adherence than monolithic deployments.
Core Intuition
Prefill is fundamentally compute-bound, demanding massive parallel throughput to process thousands of prompt tokens simultaneously. Decode is fundamentally memory-bandwidth bound, generating tokens sequentially, one at a time, relying entirely on fast reads from the KV cache. Monolithic systems compromise on both profiles. By splitting them—deploying a dedicated "Prefill Machine" and a separate "Decode Machine"—each machine's parallelism strategy, batching limits, and hardware type can be optimized specifically for its mathematical bottleneck.
Technical Deep Dive
Advanced systems like Splitwise and DistServe maintain separate, independently scaled worker pools. Prefill workers exclusively process the prompt, generate the first token, and compute the KV cache. They scale based on incoming request rates and aggregate prompt lengths. Following this, the intermediate KV cache is transmitted directly from the prefill worker to the decode worker via high-speed interconnects like NVLink (within a node) or InfiniBand (across nodes). Decode workers simply receive the KV cache and generate subsequent tokens, scaling dynamically based on the active sequence count. DistServe utilizes an online scheduling optimization algorithm to balance these parallel strategies seamlessly.