Six claims from the lesson, each tested on the Session 10 character-level GPT, with both sides of every comparison tuned. The charts below are drawn from the runs themselves, and most of them take your input.
“Adam moves every weight by approximately η, whatever the magnitude of its gradient.” (lesson §6)
Type any five gradients. The table recomputes m, v, m̂, v̂ and the step with the exact rule the notebook checked against four PyTorch implementations. Switch bias correction off and step 1 jumps to 3.16 η.
“It matters most at the first step.” (lesson §6)
For a steady gradient the uncorrected step is the corrected one times f(t) = (1 − β₁ᵗ)/√(1 − β₂ᵗ). The top recovers in about ten steps and the bottom in about a thousand, so the error grows after step 1 before it shrinks. Slide β₂ to see how long it lasts.
“That ratio, logged per layer, is the quantity to monitor. It should stay near 10⁻³ for the whole run.” (lesson §9)
All 27 weight matrices of the 6-layer GPT at every step, as ‖ΔW‖/‖W‖. Hover the heatmap to read any cell. Warmup stops changing a layer’s ratio at the first step after which the ratio stays within ±25% of the level every run ends up at.
“Cosine needs the length of the run in advance and WSD does not.” (lesson §10)
Stopping both at step 200 compares a cosine that has already decayed with a WSD that has not started decaying. So every model you could ship at step 200 is measured below, each at its own best LR, with 3 seeds.
“Under the standard parameterization it varies roughly in inverse proportion to width … under muP the minimum is at the same value at every width.” (lesson §12)
A 4-layer GPT at widths 256, 512 and 1,024, swept over 12 learning rates plus a finer second pass, with 2 seeds and 600 steps. Width 2,048 was trained but held out: the trend is fitted without it, then checked against it before anything is said about 4,096.
“Almost every optimizer claim that failed to replicate was a well tuned method measured against a badly tuned one.” (lesson §15)
Muon handles the hidden matrices and AdamW handles everything else. Muon’s step is scaled to match AdamW’s, so every arm shares one LR axis and gets the same sweep. The more carefully AdamW is tuned, the smaller Muon’s lead becomes.