How Neural Networks Learn: Gradient Descent and Backpropagation

This is the third post in our AI & ML Foundations series, building on neural networks. Start with What Is Machine Learning, Really? if you're joining partway through.

Marcus built forecasting tools for a solar energy operator — the kind of small, unglamorous prediction that quietly matters: given tomorrow's weather forecast, how much electricity will a given solar farm actually generate? Get it wrong by a lot, and the grid operator either over-buys backup power or gets caught short.

His first neural network, freshly initialized, was worse than useless — its predictions for tomorrow's output were essentially random numbers, no better than a coin flip would have been. That was expected; every weight in the network started as small random noise. What genuinely surprised him was watching those predictions get better, epoch after epoch, with no human ever touching a single line of the network's actual logic. Something inside it was changing, on its own, in response to nothing but being shown examples of past weather and past output. That something is the actual subject of this post — and it's really just two ideas working together: a way to measure how wrong the network currently is, and a way to nudge every single weight, however deeply buried, a little bit less wrong.

Step one: measuring how wrong you are

Before you can improve anything, you need a number that says exactly how bad the current prediction is. That's the loss function — for a prediction like tomorrow's energy output (a number, not a category), the simplest common choice is squared error:

def squared_error_loss(predicted, actual):
    return (predicted - actual) ** 2
 
# Model predicted 420 MWh; the farm actually generated 500 MWh
squared_error_loss(predicted=420, actual=500)
# 6400

Squaring the difference does two useful things at once: it makes the loss always positive (an overestimate and an underestimate of the same size count equally badly), and it punishes large errors disproportionately more than small ones — a prediction off by 80 MWh contributes a loss sixteen times larger than a prediction off by 20 MWh, not just four times larger. That disproportionate penalty is deliberate: it pushes training to prioritize fixing the biggest mistakes first.

Step two: figuring out which direction actually reduces the loss

Here's the part worth sitting with, because the intuition genuinely carries the whole idea. Imagine standing on a hillside in thick fog, at night, holding nothing but a device that tells you the steepness and direction of the ground under your feet, right where you're standing. You can't see the valley. You can't see the whole hill. But you can feel which direction is downhill right here, take a small step that way, and check again.

That's gradient descent, essentially in full. The "gradient" is the mathematical version of "which way is downhill from here" — computed exactly, not felt out — for every single weight in the network simultaneously. The step size is called the learning rate, and it's a real, consequential choice: too large, and you overshoot the valley entirely, bouncing between hillsides; too small, and training crawls, taking far more steps than necessary to reach a good solution. Marcus's first attempt used a learning rate ten times too large, and his loss genuinely got worse every epoch — overshooting past the valley floor and back up the far side, repeatedly, rather than settling into it.

learning_rate = 0.01
new_weight = old_weight - learning_rate * gradient

That one line — subtract the gradient, scaled by the learning rate, from the current weight — is the entire update rule. Every weight in the network, however many there are, gets updated by exactly this formula, every single training step.

Step three: getting the gradient to every weight, not just the last layer

Here's the genuinely hard problem gradient descent alone doesn't solve: Marcus's network has weights in an early layer processing raw sensor readings, a middle layer, and an output layer producing the final energy forecast. The loss is only computed at the very end. So how does a weight buried in the first layer know which direction reduces a loss that's calculated two layers later?

Backpropagation is the answer — a disciplined application of the chain rule from calculus, computing how much the final loss changes in response to a small change in the output layer first, then using that to compute how much it changes in response to the layer before it, and so on, working backward through the network one layer at a time.

The genuinely important practical point: backpropagation computes the gradient for every weight in the entire network, in a single backward pass — not one painstaking calculation per weight. For a network with millions of parameters, that efficiency is the difference between training finishing in minutes and never finishing at all.

Watching it happen, one step at a time

import tensorflow as tf
 
x = tf.Variable(3.0)  # a single toy weight, for illustration
 
with tf.GradientTape() as tape:
    loss = (x - 5) ** 2  # loss is smallest when x = 5
 
gradient = tape.gradient(loss, x)
print(gradient.numpy())  # -4.0 — the slope at x=3
 
learning_rate = 0.1
x.assign_sub(learning_rate * gradient)
print(x.numpy())  # 3.4 — one step closer to 5

GradientTape is TensorFlow's mechanism for automatically computing exactly the kind of backward-flowing gradient described in Figure 2 — for one toy variable here, but the identical mechanism scales to every weight in Marcus's real forecasting network. Run this update repeatedly, and x creeps steadily toward 5, the point where the loss is actually minimized — a single-variable version of exactly what happened, across every weight simultaneously, once Marcus fixed his learning rate and let training run.

Why training happens in batches, not one example at a time

A real detail worth knowing: a network like Marcus's doesn't compute the gradient from one day's weather data at a time — it processes a batch (commonly 32 or 64 days) and averages their gradients before taking a single step. A gradient computed from just one day is noisy and can point in a genuinely unhelpful direction due to that one day's own quirks (a freak cloud burst, a sensor glitch); averaging across a batch smooths that noise out, producing a more reliable downhill direction — the practical reason model.fit() accepts a batch_size parameter at all, rather than always training on one example at a time.

What to actually remember from this post

• A loss function turns "how wrong is the model" into a single number — squared error specifically punishes larger mistakes disproportionately more, pushing training to fix the biggest errors first.
• Gradient descent repeatedly takes a small step in the direction that locally reduces loss — guided only by the slope right where the weights currently are, with no view of the whole landscape.
• The learning rate is a genuine trade-off: too large overshoots the valley entirely; too small crawls unnecessarily slowly.
• Backpropagation is what gets the gradient to every layer, not just the last one — a single efficient backward pass through the network, using the chain rule.
• Batching averages gradients across several examples before each update, smoothing out the noise any single example would introduce on its own.

Next in the series: Overfitting, Underfitting, and the Bias-Variance Tradeoff, where a model that trained beautifully turns out to have quietly memorized the wrong thing.