The architecture.
From the language-backbone mechanisms in the Kimi K3 paper to the complete PyTorch model. Each chapter connects an idea to its mathematics, implementation, and validation.
KDA / MLA
Dense / MoE
Conceptual assembly, not a literal operation-by-operation graph. Detailed explanations, equations and evidence belong in individual chapters.
How can a model remember a long sequence without keeping everything in its working memory?
A compact memory that can correct itself, with occasional global attention when the model needs to look back.
Read chapter →What if a layer could choose which earlier ideas to bring forward?
How Kimi K3 lets a layer revisit earlier representations instead of accepting one running sum.
Read chapter →Once attention has gathered the information, what actually transforms it?
Why attention needs a per-token transformation, and how SiTU-GLU keeps its strongest signals under control.
Read chapter →How can a model have hundreds of experts without consulting every one for every token?
How Kimi K3 makes room for many experts, keeps their outputs under control, and gives quieter experts a chance.
Read chapter →How do all these pieces become one model that predicts the next token?
Follow a token from its embedding through KDA, MLA, AttnRes and MoE, all the way to the next-token scores.
Read chapter →