K3KIMI K3 FROM SCRATCH
GitHub

INDEPENDENT RESEARCH ENGINEERING / 2026

Kimi K3
from scratch.

A language model is more than an architecture on paper.

I built a scaled, Kimi K3-inspired backbone in PyTorch, then engineered and completed its six-billion-token pretraining run.

Explore the architecture See what it learned
Independent implementation · Base language model · Not an official Kimi release
MODEL / 300M CONFIGURATION

The hybrid
backbone.

09 KDA 04 Gated MLA
  1. 01KDADense
  2. 02KDAMoE
  3. 03KDAMoE
  4. 04MLAMoE
  5. 05KDAMoE
  6. 06KDAMoE
  7. 07KDAMoE
  8. 08MLAMoE
  9. 09KDAMoE
  10. 10KDAMoE
  11. 11KDAMoE
  12. 12MLAMoE
  13. 13MLAMoE
INPUT → 13 DECODER BLOCKS → OUTPUTHow the blocks fit together
294.7MTotal parameters
~100.4MActive per token
6.0BProcessed token positions
183,106Successful optimizer updates

Figures refer to this reduced-scale experiment, not the official Kimi K3 model. Methods and evidence ↗

01 / THE WORK

BEYOND THE PAPER

The interesting part was
making it all work.

A model can look convincing in a diagram long before it can survive a real training run. This project followed the engineering questions all the way through.

01

Implement

Translate KDA, gated MLA, AttnRes and Stable LatentMoE into a working scaled backbone.

02

Verify & optimize

Test equivalence, build the training system and qualify execution on an H100 NVL.

03

Train & recover

Complete the planned token budget with restorable checkpoints and auditable monitoring.

04

Evaluate

Inspect next-token prediction and actual generations—without mistaking them for a benchmark.

The backbone follows published research mechanisms, but the scaled implementation and training integration are independent. The accelerated KDA CUDA path reuses upstream Flash-Linear-Attention rather than an original kernel.

Follow the engineering journey

02 / THE EVIDENCE

A finished run.
An honest result.

The model learned recognizable language structure, but its generations still reveal repetition and drift. The outcome is a trained research backbone, not a conversational assistant.

Read the evaluation
POST-TRAINING / NEXT-TOKEN EVALUATION
21.14Perplexity
3.0510NLL · nats/token
42.27%Top-1 accuracy

64 preselected windows · 65,536 validation token targets. Different from the training monitor; not an external benchmark or baseline comparison.

03 / EXPLORE

FOUR WAYS INTO THE PROJECT

Follow your curiosity.

01 / THE MODELArchitectureThe mechanisms, their mathematics, and the PyTorch implementation.Explore Architecture 02 / THE ENGINEERINGTrainingFrom component checks to a completed, recoverable pretraining run.Explore Training 03 / THE EVIDENCEResultsMeasurements, generations, methodological limits, and what remains unknown.Explore Results 04 / THE CODEDocsA file-by-file map of the model, the trainer, and supporting scripts.Explore Docs
BUILT TO BE INSPECTED

Every layer has code. Every run has a method. Every result has a limit.

Source on GitHub ↗About the author ↗