Atlas of MachineIntelligence
Being an interactive account of how machines learn — from the first regression to the frontier of research. Set out in explainers, moving figures, exercises & papers decoded, for the newcomer and the expert alike.
- Callout 1: The model. Its weights, rolling downhill.
- Callout 2: The deepest valley. The global minimum.
- Callout 3: A lesser valley. A local minimum — a common trap.
- The rule it follows w ← w − η∇L(w)
On the Learning Rate
Imagine standing on a foggy hillside, trying to reach the valley floor. You cannot see far, but you can feel the slope beneath your feet. Gradient descent does exactly this: feel the slope, take a step downhill, repeat. The learning rate is the length of your stride — too short and you will be walking all day; too long and you leap clean over the valley and land on the other side.
The learning rate η scales every update, w ← w − η∇L, and it is usually the first hyperparameter worth tuning. Too low and training stalls; too high and the loss oscillates or diverges. Modern training rarely holds it fixed: a short warm-up, then a cosine or step decay, paired with an optimizer such as Adam that adapts the step for each parameter.
For a quadratic loss with Hessian H, gradient descent converges exactly when 0 < η < 2/λmax(H), at a rate set by the condition number κ = λmax/λmin. Neural networks are not quadratic, and full-batch training behaves strikingly: sharpness rises until λmax ≈ 2/η, then hovers there while the loss keeps falling, non-monotonically — the so-called edge of stability.
The Contents
Each chapter is a plate of related ideas. Open one at any depth.