Architecture

What is new in Moonshot AI's Kimi K3, part by part, and how each part differs from DeepSeek-V3, Qwen3, Llama 4 and gpt-oss: hybrid KDA and MLA attention for a million-token context, a latent expert layer, quantile load balancing, attention across depth, a soft-capped activation, a per-head optimizer, and training in the serving number format.

Sixteen questions about sparse models, each one the thing you would ask after hearing the last answer: what the router really is, why experts do not learn topics, why sparsity stops paying once you batch, and what it costs to serve one.