Greedy decoding48 TOKENS
, there lived a man named Ahab. He was a man of great strength and courage, and he was a man of honor. He was a man of honor, and he was a man of honor. He was a man of honor
03 / THE EVIDENCE
Six billion tokens.
What did we learn?
Building a language model is one question. Training it all the way to the planned budget is another. But the most interesting question comes afterward:what can the finished model actually do? Here is what the measurements, training logs, and the model's own words tell us.
294.7MTotal parameters
6.0BProcessed token positions
183,106Successful optimizer updates
0Logged skipped updates
01 THE RUN
The model kept learning.
And the run finished.
The first 80 million tokens were a rehearsal. The pilot's monitored validation loss fell from 7.8979 to 4.1003, enough to justify continuing with the same model and optimizer state. Then came the longer experiment: three training sessions, an interruption and recovery, and a final checkpoint at exactly 183,106 successful updates.
The curves tell a quieter story than a single final number. Loss improved quickly at the beginning, then more gradually as the model saw billions of token positions. The final monitored validation loss was 3.1825, compared with a best observed value of 3.1589 near the end. A small late rise is worth noting, but these observations alone cannot tell us whether an earlier checkpoint would generalize better.
The full audit found zero skipped updates and no recorded nonfinite gradient-norm events. Gradient clipping was applied to 2,440 of the 183,106 updates (1.33%). Its frequency increased late in training, a diagnostic signal that deserves investigation in any future run, not proof that this run failed. The measured continuation averaged about 67,056 training tokens/son the H100 NVL, excluding evaluation, saving, and interruptions.
A completed run shows that the engineering held together. It does not, by itself, tell us how well the language model generalizes. For that, we need to stop reusing the monitoring prefix.
02 THE SCORE
A fresh view of the
final checkpoint.
During training, validation checked the same short prefix again and again. The final evaluation asked a different question: how well does the frozen checkpoint predict tokens in 64 preselected, separated windowsoutside that repeatedly monitored prefix? Each window contains 1,024 next-token targets, giving 65,536 scored predictions in total.
Negative log-likelihood3.051nats / token
Perplexity21.14exp(mean NLL)
Top-1 accuracy42.27%next-token predictions
Measurement conditions. Final 294,676,808-parameter checkpoint; FineWeb-Edu prepared validation split; SmolLM2-135M tokenizer; 64 × 1,024 fixed shifted targets; local RTX 4050, FP32, PyTorch chunkwise KDA, zero state at each window. The published numbers were recomputed from the 64 per-window receipts, not from a new forward pass. Data: 64 fixed-window evaluation records and the accompanying audit summary, available in the evidence archive.
This is evidence of next-token prediction learning, not a leaderboard score. The windows come from the validation split, not a separate external test set. Moreover, training used BF16 with a fused KDA kernel, while this evaluation used the FP32 reference path. Numerical parity between those paths has not been established, so the final NLL cannot be compared directly with the training monitor's validation loss. There is no matched external baseline here.
03 THE TEXT
A sentence can sound right.
Until it doesn't.
Scores are useful, but text exposes a different side of the model. Six fixed English prompts were continued twice: once greedily, choosing the most likely next token, and once with seeded sampling. Every continuation is exactly 48 generated tokens long. Here is one unedited pair.
SHARED PROMPT
Once upon a time, in a small village
Seeded sampling48 TOKENS
, there stood a woman, her name being Kola, a woman in uniform, in a round-rigged suit, in a white robe and a white cloak. A young woman's costume, of gold-roled or j
Both continuations pick up the tone of a story. The greedy one falls into an obvious loop. Sampling escapes the repeated phrase but wanders into awkward, unfinished description. The cutoff is intentional: neither completion was allowed to run beyond the fixed 48-token limit.
Across this small set, the model learned grammar, familiar phrasing, and local genre cues. But more variety did not guarantee better meaning: sampled outputs also included topic drift and factual errors. In particular, we did not run a calibrated human-quality study, and these 48-token English snippets say little about sustained coherence or reasoning. This is a base language model, not an instruction-tuned conversational assistant.
04 WHAT REMAINS
What worked—and
what we still don't know.
The result is not a reproduction of the full-size Kimi K3's capabilities. It is a smaller, independently implemented language backbone that survived a complete pretraining run, resumed from saved state, and produced measurable next-token learning. That is a meaningful research-engineering result.
The gap between recognizable English and dependable language ability is just as real. One checkpoint, one fixed-window evaluation, and twelve short completions cannot establish factual reliability, broad reasoning, long-context quality, or competitiveness with other models. Those are still open questions, not results waiting for a more flattering description.
For now, the right decision was to stop this run and preserve its evidence. More GPU time would be a new experiment, justified by a specific hypothesis and an evaluation protocol—not an automatic extension of the same training job.