Kv-cache

Modern LLM inference is memory-bound, and the KV cache is the reason. This report walks through why the cache exists, how large it gets, the attention changes that shrink it (GQA, DeepSeek MLA), how engines like vLLM and SGLang manage it, how quantization and eviction compress it, and how disaggregated serving moves it between machines.

What happens inside a serving engine: why the GPU fills up with something other than the model, what paging the cache buys you, why reading a prompt and writing an answer fight each other, and which knobs settle the fight.