Llm

What is new in Moonshot AI's Kimi K3, part by part, and how each part differs from DeepSeek-V3, Qwen3, Llama 4 and gpt-oss: hybrid KDA and MLA attention for a million-token context, a latent expert layer, quantile load balancing, attention across depth, a soft-capped activation, a per-head optimizer, and training in the serving number format.

A long-running agent has to keep learning without forgetting what made it useful. This write-up covers catastrophic forgetting and how it is measured, the four families of fixes, memory in token space, continual RLVR, the production data flywheel, its guardrails, the multi-LoRA serving shape, and how to run the same loop with Claude.

Post-training has moved from imitating human labels to learning from verifiable outcomes. This write-up covers the RLVR objective, GRPO and the choice of KL penalty, advantage collapse and the methods that recover the lost gradient, reward hacking and noisy verifiers, outcome versus process reward models, the production data flywheel that turns failures into weight updates, long chain-of-thought and latent reasoning, and the OpenRLHF stack that runs it all.

Modern LLM inference is memory-bound, and the KV cache is the reason. This report walks through why the cache exists, how large it gets, the attention changes that shrink it (GQA, DeepSeek MLA), how engines like vLLM and SGLang manage it, how quantization and eviction compress it, and how disaggregated serving moves it between machines.

Sixteen questions about sparse models, each one the thing you would ask after hearing the last answer: what the router really is, why experts do not learn topics, why sparsity stops paying once you batch, and what it costs to serve one.