← Back
Published on

Kimi K3 Architecture Teardown

View .md

 

Scale: Total vs Active Parameters and Serving Economics

Kimi K3 is Moonshot AI's open-weight frontier model, released in July 2026. It is a sparse Mixture of Experts in the same family as DeepSeek-V3, Qwen3 and Kimi K2, and about three times the size of its predecessor. Size is the least interesting part. Almost every component of the stack has been replaced, and the report ties each replacement to a problem that appears when you train at this scale.

This post is about those replacements: what each one is, what the other open models do instead, and why Moonshot changed it. If MoE routing is new to you, start with the Mixture of Experts write-up.

Visual 1: The architecture, redrawn from Figure 2 of the Kimi K3 technical report. Right: the stack, with Attention Residuals in red reading across blocks. Top left: one Stable LatentMoE layer. Bottom left: one KDA layer.

PartKimi K3The common choice
Attention3 KDA layers to 1 Gated MLA, no positional encoding in MLAMLA everywhere (DeepSeek-V3, K2); grouped-query attention (Qwen3, gpt-oss)
Expert layerexperts run in a latent space, RMSNorm on their sumfull-width fine-grained experts, sum added as is (DeepSeek-V3, Qwen3, K2)
Load balancingbias set from a score quantilebias nudged by a fixed step (DeepSeek-V3, K2); auxiliary loss (Qwen3)
Residual pathattention over earlier blocksa running sum of every layer (all of them)
ActivationSiTU-GLU, soft-cappedSwiGLU, unbounded (all of them)
OptimizerPer-Head Muon with weight clippingAdamW (DeepSeek-V3); MuonClip (K2)
NumericsMXFP4 experts and MXFP8 activations in the training loopFP8 training (DeepSeek-V3); MXFP4 applied after training (gpt-oss)

Sparsity itself is not new. Like DeepSeek-V3, K3 runs a small fraction of its weights per token, so the serving cost is memory and network traffic rather than arithmetic. What is new on the serving side is the cache. KDA keeps a fixed-size state instead of a growing KV cache, and Mooncake's prefix cache checkpoints that state, so a repeated prompt resumes prefill from the hit point. That is why a cached input token costs a tenth of a fresh one.

Stable LatentMoE vs DeepSeekMoE

DeepSeekMoE set the expert layout every open MoE now uses: many thin experts a token can combine, plus a shared expert every token passes through. K3 keeps that layout and changes two things. The routed experts run in a latent space at half the model width, which halves their cost and is what pays for 896 of them per layer. And their weighted sum passes through an RMSNorm before it rejoins the shared branch, so the routed branch always arrives at one fixed scale.

In DeepSeek-V3 and Qwen3 that sum is added as is, and its size swings with which experts fired. The report names runaway activations as the failure the norm prevents.

Visual 2: One Stable LatentMoE layer. The routed branch runs in the latent and gets one fixed scale. The shared experts never leave full width.

Load Balancing: Quantile Balancing vs Bias Terms

Every MoE has to stop a few experts from winning all the traffic. Qwen3 uses an auxiliary loss, which fights the language modeling loss. DeepSeek-V3 and K2 use a bias per expert, nudged by a fixed step each batch, which leaves a step size to tune. K3 sets the bias directly: it keeps a histogram of each expert's routing margins and reads off the quantile that gives the expert exactly its target share. No loss term, no step size, and the load is set in one update rather than chased.

Visual 3: The report's toy example. Loads of 4, 3, 1, 0 become 2, 2, 2, 2 in one update.

Hybrid Attention for 1M Tokens: KDA and Gated MLA

The attention is where K3 departs furthest from the field. DeepSeek-V3 and K2 use MLA in every layer, Qwen3 and gpt-oss use grouped-query attention, and all of them keep a KV cache that grows with the context. K3 uses Kimi Delta Attention, a linear attention with a fixed-size state per head, in three layers out of four, and Gated MLA in the fourth. The state does the long spans cheaply. The MLA layers do exact lookup across the whole context.

Visual 4: The 3:1 pattern, and what each kind of layer keeps in memory.

KDA's state is rewritten once per token: decay every channel a little, erase part of what was stored under the current key, write the new value. Two details are K3's own. The decay is floored so a 16-token tile stays in BF16 range and runs on Tensor Cores, which earlier linear attention could not do. And the MLA layers use no positional encoding at all, a move Llama 4 also made in some layers. Position comes from the KDA decay around them, so nothing is stretched past its training length, which is where long contexts usually fray.

Visual 5: One KDA step. The state is the same size after a million tokens as after ten.

Attention Residuals: Depth-wise Attention Instead of Sums

Every Transformer since the original, DeepSeek and Qwen included, adds each layer's output to a running sum. The sum keeps growing with depth, so deep layers have to shout to be heard. Moonshot's Attention Residuals paper calls this PreNorm dilution, and K3 ships the paper's alternative: each block learns a query, scores the earlier blocks against it, and takes a weighted mix, so a deep layer can read the embedding directly. The Block variant, 8 blocks of 12 layers, keeps the cost at nine stored summaries instead of 93.

Visual 6: Three residual designs. K3 uses the third.

Training Stability at 3T Scale

Most of the remaining changes bound something the standard recipe leaves unbounded. The optimizer is Per-Head Muon: Muon's orthogonalized updates, split per attention head so no head's gradients dominate, with K2's weight clipping to keep attention logits in range. DeepSeek-V3 trains with AdamW. The activation is SiTU-GLU, a tanh soft-cap on both branches of the feed-forward block, where everyone else uses unbounded SwiGLU.

The number format is part of training too. K3 is fine-tuned and reinforced with MXFP4 expert weights and MXFP8 activations in the loop, where DeepSeek-V3 trains in FP8 and gpt-oss quantizes to MXFP4 after the fact. The model that ships is the model that was trained.

Visual 7: Five bounds, from the optimizer down to the serving format.

Post-Training: Long-Horizon Demos, Not Benchmarks

Post-training follows the now-common recipe: supervised fine-tuning, reinforcement learning, and distillation back into one model. The report's distinctive move is to show results as three long projects rather than only scores.

DemoWhat K3 builtWhat the report says
MiniTritona Triton-like compiler: Python tile frontend, MLIR layer, PTX code generationtrains a small GPT end to end; built with a maintainer's direction and review
Kernel optimizationa faster kernel for its own Attention Residualslatency cut by more than half within a 24-hour budget; the one task scored against other models
Chip designan inference-chip prototype for a nano model, through open EDA toolsa single 48-hour autonomous run; timing closed in RTL simulation

Only the kernel task is scored against other models, so read the compiler and the chip as demonstrations.

What We Learned

  • K3 is a redesign, not a scale-up. Seven parts changed, each tied to a failure at this size.
  • Attention is the big departure. A fixed-size state in three layers of four, exact lookup in the fourth, no positional encoding.
  • Experts moved into a latent space and got a norm. DeepSeek's layout, made cheaper and steadier.
  • Balance is set, not chased. The bias comes off a quantile, with no loss term and no step size.
  • Everything else got a bound. Attention over depth, a per-head optimizer, a capped activation, and training in the serving format.

Sources and further reading