KV Cache Explained: Prefill, Decode, Context Windows & Why LLMs Need So Much Memory

Imagine you have just purchased a high-end GPU with 24GB of VRAM. You load a well-optimized 8B model; the weights take up a modest 16GB, leaving 8GB to spare. Everything runs smoothly. Then you paste in a long PDF, or continue an hour-long coding session with hundreds of lines of context. Your system freezes. The terminal outputs a fatal error: Out of Memory. Your application crashes.

You did not load any new model weights. The neural network did not grow larger. So where did those 8 gigabytes go?

The answer is the KV cache, the most misunderstood component in LLM inference. Understanding it explains nearly every puzzling behavior in local AI: why the first token takes longer than expected, why long conversations gradually slow to a crawl, and why context windows have physical memory limits rather than just software ones.

What Is the KV Cache?

The KV (Key-Value) Cache is a dynamic, temporary memory bank that LLMs use to speed up text generation. As a model processes your prompt and generates a response, it performs billions of calculations to understand how every token relates to every other token: a mechanism called self-attention.

Without optimization, the model would recalculate these relationships from scratch for every new token it generates. For a conversation with 5,000 tokens, generating the 5,001st token would require recomputing the relationships across all 5,000 previous tokens. This creates an O(N²) computational problem that would make practical AI inference prohibitively expensive.

The KV cache solves this by saving the intermediate results: the "Keys" and "Values" from the attention mechanism, in GPU memory. Instead of recomputing history, the model looks it up. The tradeoff: speed at the cost of VRAM.

The Real Cost of Generating a Token

A language model is not a brain that reads sentences. It is a deterministic mathematical engine that performs massive matrix multiplications to predict one token at a time. Here is what happens for every single token generated:

1. Tokenization: Your text is split into tokens (subword units each assigned a numeric ID. "Hello, world" becomes [Hello] [,] [world].)
2. Embeddings: Each token ID is mapped to a high-dimensional vector representing its meaning in the model's learned concept space.
3. Transformer Layers & Self-Attention: Inside each layer, every token computes how much "attention" it should pay to every other token. The word "bank" in "the bank of the river" learns to attend strongly to "river" rather than "money".
4. Output Distribution: After all layers, the model produces a probability score over its entire vocabulary (tens of thousands of tokens) and selects the highest-scoring candidate.

This entire process runs once per token. A 500-word response requires hundreds of iterations of this pipeline. The KV cache exists to make that repetition practical.

Prefill vs Decode

LLM inference has two fundamentally distinct phases with different hardware bottlenecks. Nearly every performance behavior (slow startup, sluggish generation, OOM errors at long context) traces back to understanding how these phases interact.

The LLM Inference Pipeline: Prefill processes the full prompt in parallel; Decode generates tokens sequentially using the KV cache.

Why the First Token Is Slow

The noticeable pause after hitting Enter, before the AI starts typing, is called Time to First Token (TTFT). It is a direct consequence of the Prefill bottleneck.

During Prefill, every token must mathematically compare itself against every previous token. The workload scales quadratically:

This is why RAG (Retrieval-Augmented Generation) architectures retrieve only the most relevant chunks rather than feeding entire documents into the prompt, reducing Prefill cost dramatically.

What Is Actually Stored in the KV Cache

Inside each transformer layer, the self-attention mechanism uses three components: Query (Q), Key (K), and Value (V), to determine how tokens should relate to one another.

Think of a conference room. You are searching for a database expert (your Query). Everyone wears a name tag listing their expertise (their Key). When your Query matches someone's Key, you listen to their contribution (their Value).

When the model processes a token, it computes Q, K, and V for that token. Here is the key insight: the Keys and Values of past tokens never change. Only the new token needs a new Query. So the model stores the K and V matrices for every prior token in VRAM, and each new Query simply looks up against that growing database. This is the KV cache.

KV Cache vs Model Weights

Confusing these two is the most common mistake in LLM memory planning. They are fundamentally different in nature:

This distinction matters in production: if you host a 70B model, the weights are shared across all users. But a user uploading a 50,000-word document consumes gigabytes of VRAM for their unique KV cache, while a user asking a one-line question consumes almost nothing. Cache memory scales per-user; weight memory does not.

Why LLMs Need So Much Memory

VRAM consumption during inference comes from four distinct sources. For most developers, only the first two matter in practice:

Where LLM memory goes: weights are the fixed cost, the KV cache is the variable one that grows with every token.

1. Model Weights: The AI's permanent knowledge. A fixed cost determined by model size and quantization. A 32B model at 4-bit uses roughly 20GB regardless of conversation length.
2. KV Cache: The AI's active working memory. Dynamic, scales linearly with the number of tokens in your context window. This is what causes OOM errors mid-conversation.
3. Activations: Temporary tensors used during the forward pass. These spike massively during Prefill (when all prompt tokens are processed simultaneously) and then largely disappear during Decode.
4. Runtime Overhead: Memory consumed by the inference engine (llama.cpp, vLLM, Ollama) for CUDA contexts, memory fragmentation buffers, and batching infrastructure. Typically 0.5–2GB depending on the engine.

Context Window Memory Costs

Every token in your prompt and every token the model generates adds a new Key-Value entry to the cache. The context window limit is not an arbitrary software setting; it reflects the physical memory available for the cache to grow into.

KV cache memory grows linearly with context length. At 128K tokens, the cache for an 8B model exceeds the model weights themselves.

How Much Memory Does One Token Use?

For Llama 3 architecture models using GQA and 16-bit precision:

The formula: 2 (K+V) × 2 bytes (fp16) × layers × kv_heads × head_dim. For Llama 3 8B: 2 × 2 × 32 × 8 × 128 = 131,072 bytes per token.

How the Cache Scales with Context

At 128K context, the Llama 3 8B KV cache (17GB) exceeds the model weights themselves. Running Llama 3 70B at full 128K context requires budgeting over 40GB of VRAM purely for conversation memory.

What Context Window Can Your Hardware Handle?

Practical context limits when running a quantized 8B-class model (~5GB weights) and reserving room for the KV cache:

Why Long Conversations Become Slower

A 20-message chat feels snappy. A 500-message chat feels sluggish. The mechanism is straightforward once you understand Decode phase dynamics.

During Decode, token generation speed is bounded by memory bandwidth. To produce each new token, the GPU must not only load the model weights but also stream the entire KV cache through its compute cores to calculate attention scores over all prior tokens.

As your conversation grows to 30,000 tokens, the KV cache might occupy 10GB. To generate each token, the GPU must read that 10GB across the memory bus. At 20 tokens per second, that means attempting 200GB of cache reads every second, quickly hitting the physical memory bandwidth ceiling. Token generation speed measurably degrades as the cache grows.

Flash Attention vs KV Cache

These two optimizations are often confused because both relate to the attention mechanism. They solve entirely different problems:

Flash Attention and KV Cache complement each other: Flash Attention makes long Prefill feasible, KV Cache makes long Decode practical. Modern inference engines use both.

Prefill vs Decode by Use Case

Depending on what you are building, the hardware bottleneck shifts entirely. Optimizing the wrong phase wastes infrastructure spend:

KV Cache Compression

Because the KV cache grows so large, AI researchers have developed dedicated compression techniques. Note that standard model quantization (GGUF, AWQ, GPTQ) compresses model weights; it does not compress the KV cache, which defaults to 16-bit precision unless explicitly configured otherwise.

KV Cache Quantization

The most practical technique for local AI users. By reducing the numerical precision of stored Keys and Values:

TurboQuant

A more advanced technique from Google DeepMind, presented at ICLR 2026. TurboQuant uses a two-step approach: first, a randomized Hadamard transform spreads outlier values across dimensions, making quantization easier. Second, the Quantized Johnson-Lindenstrauss (QJL) transform removes the bias this introduces. The result is KV cache compression to 3-bit precision with near-zero accuracy loss, achieving roughly 6x memory reduction, making 128K+ context windows practical on consumer hardware that previously could not support them. Early llama.cpp community implementations are already available. See our detailed TurboQuant breakdown for a full explanation.

Why This Matters for Local AI

If you run models locally with Ollama, llama.cpp, LM Studio, or Open WebUI, the KV cache directly determines your stability, speed, and maximum context length.

• Ollama: The num_ctx parameter in your Modelfile controls the pre-allocated context window size. Setting it to 128K on a 16GB machine will cause immediate OOM. Match this value to your hardware. For 16GB, 4K–8K is a safe starting point for most models.
• llama.cpp: The -c flag defines the pre-allocated KV cache size. Enable 8-bit KV caching with -ctk q8_0 -ctv q8_0; this is one of the highest-impact single-flag optimizations available and is particularly valuable on MacBooks and 24GB GPUs.
• LM Studio: Provides a "Context Length" slider and options to enable Flash Attention and KV cache quantization. Do not slide context length to the maximum without checking your available headroom; the UI does not warn you before an OOM crash.
• General rule: If you experience a model that starts fast but degrades over a long session, the KV cache is the culprit. Either reduce num_ctx, enable cache quantization, or use a smaller model to free more headroom.

Visual Walkthrough: One Prompt, Start to Finish

Here is what actually happens when you type: "Write a poem about a cybernetic forest."

Common Misconceptions

Frequently Asked Questions

What is the KV cache in simple terms?

The KV cache is a temporary memory bank where an LLM saves its intermediate mathematical results: the "Keys" and "Values" of the attention mechanism, for every past token. By storing these, the model avoids recomputing the entire conversation history each time it generates a new word, trading VRAM for speed.

What is the difference between Prefill and Decode in LLM inference?

Prefill is the initial reading phase where the model processes your entire prompt in parallel; it is compute-bound (limited by TFLOPS) and determines Time to First Token. Decode is the generation phase where the model produces one token at a time using the KV cache; it is memory-bandwidth-bound (limited by GB/s) and determines how fast text streams to your screen.

Why is the first token slow?

Before generating any output, the model must process your entire prompt simultaneously during Prefill. Every token compares itself against every other token, scaling quadratically. A 10,000-token prompt requires millions of matrix multiplications, creating a noticeable delay before the first word appears. This is called Time to First Token (TTFT).

Which consumes more memory: model weights or the KV cache?

For short conversations, model weights dominate. For long contexts, the KV cache often overtakes them. A Llama 3 8B model at 128K context requires roughly 17GB for the KV cache alone, more than the model weights. In production inference systems with many concurrent users, the cache is frequently the single largest memory consumer.

Why does VRAM usage increase during a conversation?

Every token you send and every token the model generates gets appended to the KV cache as a new Key-Value pair. The longer the conversation, the physically larger the cache grows in VRAM. Once the cache fills your remaining VRAM, the system either crashes or falls back to much slower system RAM offloading.

Why does a model fit in memory initially but crash later?

You successfully loaded the Model Weights into VRAM but left insufficient headroom for the KV Cache to grow. The weights are a fixed cost; the cache is dynamic. As the conversation lengthens, the cache fills the remaining VRAM until the system crashes. Always leave 20–30% of VRAM empty after loading a model.

What is Grouped Query Attention (GQA) and why does it matter?

GQA is an architectural optimization where multiple Query heads share a single Key and Value head. In older Multi-Head Attention models, every head stored its own K-V pairs, causing the cache to explode in size. GQA reduces the number of stored K-V pairs by up to 8x with minimal quality loss. Most modern models including Llama 3, Qwen 3, and Mistral use GQA, which is a major reason their cache requirements are manageable.

What is the difference between Flash Attention and KV Cache?

They solve different problems. Flash Attention reduces the temporary activation memory used during Prefill by computing attention in blocks rather than materializing a full N×N matrix. KV Cache reduces redundant recomputation during Decode by storing and reusing past Key-Value pairs. Flash Attention saves Prefill memory; KV Cache saves Decode compute.

Why does Ollama become slower during long chats?

As the KV cache grows, the GPU must load the entire cache across its memory bus to calculate attention for each new token. A 30,000-token conversation might produce a 10GB KV cache. At 20 tokens per second, the GPU must read that 10GB cache 20 times every second, quickly saturating memory bandwidth and causing measurable slowdowns.

Does quantization reduce KV cache size?

Standard weight quantization (GGUF, AWQ, GPTQ) shrinks model weights but does not affect the KV cache, which defaults to 16-bit precision. To reduce cache size you must explicitly enable KV cache quantization: 8-bit halves the memory requirement, 4-bit reduces it by 75%. In llama.cpp, use -ctk q8_0 -ctv q8_0 to enable 8-bit KV caching.

Can KV cache be stored in system RAM instead of VRAM?

Yes, this is called offloading. However, system RAM bandwidth is far slower than GPU VRAM, which means the memory bus bottleneck during Decode becomes dramatically worse. A model generating 30 tokens per second from VRAM may drop to 1–3 tokens per second when the cache is offloaded to system RAM.

Does clearing a chat delete the KV cache?

Yes. The KV cache is tied to your active conversation session. When you start a new chat or clear the context, the VRAM allocated to that cache is freed. This is why starting a fresh conversation often immediately improves performance after a long sluggish session.

What is TurboQuant and how does it help?

TurboQuant is a KV cache compression technique from Google DeepMind, presented at ICLR 2026. It applies a randomized Hadamard transform followed by the Quantized Johnson-Lindenstrauss (QJL) method to compress the cache to 3-bit precision with near-zero accuracy loss, roughly 6x memory reduction. This makes very long context windows (128K+) practical on consumer hardware. Early llama.cpp community implementations are already available.