Gradient Descent, Visually

Most of machine learning comes down to one task: find the settings that make some error as small as possible. Gradient descent does it the way you’d get down a foggy hill. You can’t see the bottom, but you can feel which way the ground slopes under your feet. So you take a step downhill, check the slope again, and repeat.

$$\underbrace{\theta_{t+1}}_{\text{next guess}} \;=\; \underbrace{\theta_t}_{\text{current guess}} \;-\; \underbrace{\eta}_{\text{step size}}\;\underbrace{\nabla f(\theta_t)}_{\text{uphill direction}}$$
The gradient \(\nabla f\) points in the direction where \(f\) climbs fastest. Its length says how steep it is there.
The minus sign turns you around, so each step goes downhill.
The learning rate \(\eta\) sets how far you step. It turns out to matter more than anything else.

1. One step at a time

In one dimension the gradient is just the slope \(f'(x)\). The orange tangent line shows the slope at the current point. Gradient descent moves \(x\) by \(-\eta f'(x)\): left when the curve rises to the right, right when it falls, and in bigger steps where it’s steeper. Click the chart to start from somewhere else.

Height after each step

2. The step size decides everything

Same bowl, \(f(x) = x^2\), same start at \(x_0 = 2\), twenty steps each. The only thing that changes is \(\eta\).

Distance from the bottom, on a log scale

On a log scale steady convergence is a straight line going down. The steeper the line, the faster the method converges.

Why: every step multiplies \(x\) by the same number

For \(f(x)=x^2\) the slope is \(2x\), so one step gives \(x - \eta\cdot 2x = (1-2\eta)\,x\). If that factor’s size is below 1, \(x\) shrinks toward 0 and you converge. Above 1, it grows and you blow up. A negative factor flips the sign each step, which is the zig-zag. Drag across the chart to pick \(\eta\).

The same rule holds for any smooth function. Swap the 2 for the curvature \(L\) (how sharply the slope changes), and a step is safe when \(\eta < 2/L\). Sharp valleys need small steps.

3. Two dimensions: reading the contour map

With two parameters the error is a landscape. Each contour line joins points at the same height, and fainter shading means lower ground. The gradient is always perpendicular to the contours, so plain gradient descent cuts straight across them. The small arrows show \(-\nabla f\), the downhill direction. Click the map to pick a new starting point.

Gradient descent steps along \(-\eta\nabla f\). It has no memory, so in a narrow valley it zig-zags from wall to wall.
Momentum keeps a running velocity \(v \leftarrow \beta v + \nabla f\), then steps \(-\eta v\). Pushes that agree add up and pushes that flip cancel, like a ball rolling downhill.
Adam divides each coordinate by a running estimate of its typical gradient size. Every direction then takes steps of about \(\eta\), however steep or flat it is.

Loss along each path, on a log scale

Hover to find that step on the map.

4. A real fit, and why scaling your inputs matters

Here we fit a line \(\hat y = w x + b\) to 40 points by gradient descent on the mean squared error, \(L(w,b) = \tfrac1n\sum_i (w x_i + b - y_i)^2\). We run it twice. The orange run uses the raw \(x\), which runs from 2 to 10. The blue run first z-normalizes \(x\). The data and the model are the same, but the two runs see very different landscapes.

With raw \(x\), changing \(w\) affects the loss far more than changing \(b\), because every \(x_i\) is large. The bowl becomes a long, thin, tilted valley, and \(\eta\) has to stay small enough for its steep walls, so progress along the flat floor crawls. Standardizing makes the bowl nearly round, so the gradient points almost straight at the minimum. (Each map has its own range, but within a map \(w\) and \(b\) share one scale.)

Distance from the best possible fit, on a log scale

Things to try