Gradient Descent, Visually
Most of machine learning comes down to one task: find the settings that make some error as small as possible. Gradient descent does it the way you’d get down a foggy hill. You can’t see the bottom, but you can feel which way the ground slopes under your feet. So you take a step downhill, check the slope again, and repeat.
1. One step at a time
In one dimension the gradient is just the slope \(f'(x)\). The orange tangent line shows the slope at the current point. Gradient descent moves \(x\) by \(-\eta f'(x)\): left when the curve rises to the right, right when it falls, and in bigger steps where it’s steeper. Click the chart to start from somewhere else.
Height after each step
2. The step size decides everything
Same bowl, \(f(x) = x^2\), same start at \(x_0 = 2\), twenty steps each. The only thing that changes is \(\eta\).
Distance from the bottom, on a log scale
On a log scale steady convergence is a straight line going down. The steeper the line, the faster the method converges.
Why: every step multiplies \(x\) by the same number
For \(f(x)=x^2\) the slope is \(2x\), so one step gives \(x - \eta\cdot 2x = (1-2\eta)\,x\). If that factor’s size is below 1, \(x\) shrinks toward 0 and you converge. Above 1, it grows and you blow up. A negative factor flips the sign each step, which is the zig-zag. Drag across the chart to pick \(\eta\).
The same rule holds for any smooth function. Swap the 2 for the curvature \(L\) (how sharply the slope changes), and a step is safe when \(\eta < 2/L\). Sharp valleys need small steps.
3. Two dimensions: reading the contour map
With two parameters the error is a landscape. Each contour line joins points at the same height, and fainter shading means lower ground. The gradient is always perpendicular to the contours, so plain gradient descent cuts straight across them. The small arrows show \(-\nabla f\), the downhill direction. Click the map to pick a new starting point.
Loss along each path, on a log scale
Hover to find that step on the map.
4. A real fit, and why scaling your inputs matters
Here we fit a line \(\hat y = w x + b\) to 40 points by gradient descent on the mean squared error, \(L(w,b) = \tfrac1n\sum_i (w x_i + b - y_i)^2\). We run it twice. The orange run uses the raw \(x\), which runs from 2 to 10. The blue run first z-normalizes \(x\). The data and the model are the same, but the two runs see very different landscapes.
With raw \(x\), changing \(w\) affects the loss far more than changing \(b\), because every \(x_i\) is large. The bowl becomes a long, thin, tilted valley, and \(\eta\) has to stay small enough for its steep walls, so progress along the flat floor crawls. Standardizing makes the bowl nearly round, so the gradient points almost straight at the minimum. (Each map has its own range, but within a map \(w\) and \(b\) share one scale.)
Distance from the best possible fit, on a log scale
Things to try
- In section 1, pick “Two valleys” and start on the right. Gradient descent settles in the nearer, shallower valley and never finds the deeper one. It only ever sees the local slope.
- Try the plateau. Far from the middle the slope is nearly zero, so the steps are tiny. This is the vanishing gradient problem in miniature.
- Raise the stretch κ in section 3. Past \(\kappa = 2/\eta\) plain gradient descent diverges, momentum survives longer, and Adam barely notices.
- On the saddle, start exactly on the ridge (y = 0). The gradient there has no sideways part, so gradient descent slides into the saddle and stops. Start a hair off the ridge and watch which methods escape and how fast.
- In section 4, push the raw η up a little. It diverges at the largest stable η shown above its map, long before the standardized run even feels the limit.