KV Cache in LLMs
An illustrated walkthrough: prefill vs decode, toy Q·K scores, why attention recomputes Keys and Values, and how KV Cache stores and reuses them.
KV Cache — Key and Value cache — is how modern LLMs avoid recomputing attention work for tokens they have already seen.
Language models generate text one token at a time. Each new token looks back at everything before it. Without a cache, that look-back recomputes the same Key and Value vectors over and over.
Goal for this page: walk a tiny sequence (I love teaching AI), see Q / K / V with toy numbers, watch the waste without a cache, then store K and V once and reuse them during decode. Same idea production engines use — small enough to follow on one screen.
An LLM predicts the next token from everything it has seen so far — then appends it and repeats.
A token is a small piece of text (a word, a subword, or a character). For this walkthrough, treat each word as one token.
Every prediction needs the full history. That is the rule that makes KV Cache matter: the past keeps growing, but most of it does not change.
Token generation is not one uniform loop. It splits into prefill and decode — and KV Cache is the bridge between them.
Prefill
Run the whole prompt in one forward pass. Build Q, K, V for every prompt token, run attention across them, and write K and V into the cache.
Parallel over prompt length · fills the cache · produces the first new token.
Decode
For each later token: compute Q / K / V for only that token, attend against the cached past, append the new K and V, emit the next token.
Sequential · one token at a time · reuses the cache every step.
Prefill is often compute-heavy but parallel. Decode is usually memory-bandwidth heavy: you stream one Query against a growing pile of cached Keys and Values. Without the cache, decode would also recompute that pile from scratch every time.
The attention layer decides which past tokens matter for the next prediction.
Attention turns every token into three vectors — Query, Key, and Value. The current token asks a question (Q), past tokens advertise what they hold (K), and the useful ones contribute their content (V).
Imagine a classroom: a new student asks a question (Query). Existing students wear name tags for what they know (Keys). Once the right people are found, their notes (Values) are what actually get used.
Query
What the current token is looking for — “who here is relevant to what I need next?”
Computed for the token being processed right now.
Key
A name tag for each token — “I know about X.” Queries match against these tags.
Needed again for every future step → worth caching.
Value
The notes behind the name tag — the information attention pulls in when a Key matches.
Also needed again for every future step → worth caching.
Score = how well the current Query matches each Key → turn scores into weights (softmax) → mix the Values by those weights. The mixed result flows into the rest of the network and helps pick the next token.
output ≈ softmax(Q · Kᵀ / √d) · V
During generation, a token may only attend to itself and earlier tokens — never the future. That is why past Keys and Values stay valid when we append a new token: nothing earlier needs to be rewritten because of something that comes later.
Tiny 2-D vectors so you can do the arithmetic by hand. Real models use hundreds of dimensions and many heads — the algebra is the same.
Prompt tokens I and love already have Keys and Values in the cache. We are about to predict the next token. For the live position we build a Query (and we will also create K / V for whatever token we emit next).
| Past token | Score = Q · K | After softmax (toy) |
|---|---|---|
| I | 1·1 + 2·0 = 1 | ~0.27 |
| love | 1·0 + 2·1 = 2 | ~0.73 |
context ≈ 0.27·[2, 0] + 0.73·[0, 3] ≈ [0.54, 2.19]
That context vector (per head / layer) is what the model uses downstream. With a KV Cache, the Keys and Values for I and love were read, not recomputed. Only the live Query (and the new token’s K / V once we pick it) are fresh.
Softmax numbers here are rounded for the page. The point is the pipeline: cached K for matching, cached V for mixing — not the exact floats.
Without a cache, every step recomputes Key and Value for the entire sequence — not just the new token.
Tokens marked in red were already computed in an earlier step. Their Keys and Values did not change. Computing them again changes nothing — it only burns time and memory bandwidth.
Step 1 · “I love”
Compute Q, K, V for both tokens. Predict teaching.
Step 2 · “I love teaching”
Recompute I and love. Compute teaching. Predict AI.
Step 3 · “I love teaching AI”
Recompute the first three. Compute AI. Waste grows with every step.
At step t you pay for t tokens even though t − 1 of them already had correct K and V. Over a long answer that becomes quadratic-feeling work in practice: each new token redoes an ever-longer prefix.
Compute Key and Value once per token, store them, and reuse them on every later step.
When a new speaker talks, you do not ask everyone else to repeat themselves — you read your notes and listen only to the new person. The notes are the cache.
Step 1 · prefill “I love”
Compute Q, K, V for I and love. Save K and V into the cache. Predict teaching.
Step 2 · decode “teaching”
Load K,V for I and love. Compute Q, K, V only for teaching. Append teaching’s K,V. Attend (Q against all cached Keys). Predict AI.
Step 3 · decode “AI”
Load three cached pairs. Compute Q, K, V only for AI. Append AI. No recomputation of the past.
New: hidden state → Q, K, V for the latest token · attention using that Q against the full cached K · weighted sum of cached V (+ the new V) · prediction head.
Unchanged: every earlier token’s K and V stay exactly as written. The cache only grows by one slot.
“The KV Cache” is not one flat list of words — it is tensors of Keys and Values for every layer (and usually every attention head).
A Transformer has many attention layers stacked. Each layer has its own Keys and Values. Multi-head attention splits the work across heads. So the cache is roughly:
One Key vector per token per head per layer.
One Value vector per token per head per layer.
Every new token appends along the sequence axis.
Bytes ≈ 2 (K and V) × layers × heads × tokens × head_dim × bytes_per_number (often 2 for FP16). Multiply by batch size if you serve several sequences at once.
That is why long context + big batches fill GPU memory fast — even when the model weights themselves fit.
Why is it a KV cache — not a QKV cache?
The Query belongs to the current token (or, in prefill, to each position while that pass runs). Once that step’s attention is done, those Queries are finished. The next decode step builds a brand-new Query for the new token.
Keys and Values of past tokens stay useful for the whole generation: every future Query must still match against them and mix their Values. So we store K and V — and discard Q.
Cache
- Key for every past token (all layers / heads)
- Value for every past token (all layers / heads)
Do not cache
- Query — only needed for the live step
- Final logits — recomputed when needed
Side by side: growing work vs almost-constant new work after prefill.
Without KV Cache
- Step 1: K,V,Q for 2 tokens
- Step 2: K,V,Q for 3 (2 recomputed)
- Step 3: K,V,Q for 4 (3 recomputed)
- Step N: K,V,Q for N+1 (N recomputed)
With KV Cache
- Step 1: K,V,Q for 2 → save K,V
- Step 2: compute 1 new · reuse 2
- Step 3: compute 1 new · reuse 3
- Step N: compute 1 new · reuse N
Without cache: 2 + 3 + … + 100 = 5,049 K/V computations
With cache: 2 + 1 + 1 + … + 1 = 101 K/V computations
About ~50× fewer Key/Value projections on that toy length. Longer sequences widen the gap. (Attention still touches more Keys as context grows — but you are not rebuilding those Keys from the token embeddings every time.)
Faster decode costs RAM (or VRAM): the cache stores K and V for every token so far.
Longer prompts and longer answers mean a larger cache. Serving many users at once multiplies that by batch size. Systems then invent tricks — paging blocks of cache, sharing a common prompt prefix across requests, quantising K/V — because the speed win is mandatory but the memory bill is real.
For almost every interactive use case, paying memory to save compute is worth it. The engineering challenge is keeping that memory under control at scale.
A few points that often trip people up the first time they meet KV Cache.
- It is not a disk cache or Redis It lives in GPU (or accelerator) memory for the lifetime of that generation — a tensor buffer the attention kernels read and append to.
- It does not store the text tokens themselves It stores the Key and Value vectors derived from those tokens at each layer. The string “love” is not what gets reused; the vectors produced for “love” are.
- Changing the prompt invalidates the prefix If you edit earlier tokens, their Keys and Values would change, so that prefix of the cache is no longer valid. Unchanged shared prefixes can sometimes be reused across requests (prompt caching) — that is a systems optimisation on top of the same idea.
- Attention cost still grows with context Even with cached K/V, the live Query still compares against more Keys as the sequence lengthens. KV Cache removes reprojection waste; it does not make attention free.
Takeaway
KV Cache is a memo pad for attention: write each token’s Key and Value once (during prefill or when the token is first decoded), then let every later Query read them instead of rebuilding the past. That is why decode can stay fast as the sequence grows — and why GPU memory becomes the bill you pay for that speed.
prefill prompt → fill KV Cache → for each new token compute Q/K/V once → append K,V → attend with Q against cached K → mix cached V → emit token → repeat.