Building micrograd to understand neural network training

Neural networks can seem mysterious because a modern training system has many moving parts. Andrej Karpathy starts with a smaller question: if a prediction and its error are built from elementary calculations, how can we tell which way to change each weight? His answer is to build micrograd, a scalar automatic differentiation engine, then use it to train a tiny network.

A forward pass evaluates an expression. Backpropagation walks its computation graph backward, using the chain rule to calculate how the final output responds to each intermediate value and input. The same machinery applies to mathematical expressions beyond neural networks. Micrograd handles individual numbers for clarity; production systems package numbers into tensors and use parallel operations for efficiency. The lecture's code sizes and repository details describe its 2022 snapshot.

Consider f(x) = 3x² − 4x + 5. At x = 3, the output is 20. Increase x by a small amount h, recompute the output, and divide the change in output by h: (f(x+h) − f(x))/h. The exact derivative is the limit f′(x) = lim(h→0) (f(x+h) − f(x))/h; for this function it is 6x − 4, exactly 14 at 3. With a finite h, the quotient is an approximation to that slope. At −3 the slope is negative; at 2/3 it is zero. Making h arbitrarily tiny is not a cure-all: finite-precision arithmetic eventually harms the estimate.

11:53View full image

With several inputs, change one while holding the others fixed. For d = a*b + c, with a = 2, b = −3, c = 10, the output is 4. The partial derivatives are ∂d/∂a = −3, ∂d/∂b = 2, and ∂d/∂c = 1. Each describes a local response, not what happens after an arbitrarily large change.

For example, raising a by h = 0.0001 while holding b and c fixed lowers d from 4 to 3.9997. Dividing the change by h gives approximately −3: because b is negative, increasing a makes their product more negative. Repeating the experiment for b and c gives the other two sensitivities. This numerical check estimates a derivative; it does not replace the exact local rules used in the engine.

Karpathy wraps each scalar in a Value object with its numerical data, a grad initially set to zero, and links to the values and operation that produced it. Overloaded addition and multiplication create new Values while recording their parents. A graph drawing can show both value boxes and operation nodes; the latter are visualization aids, not additional Values.

For L = (a*b + c)*f with a = 2, b = −3, c = 10, and f = −2, the intermediate values are a*b = −6, d = 4, and L = −8. The graph preserves the sequence needed to trace the output back to the inputs.

Each node's grad represents the derivative of the final output with respect to that node. Start with ∂L/∂L = 1. At L = d*f, the local derivative with respect to d is f = −2, and the local derivative with respect to f is d = 4. An addition passes its incoming gradient unchanged to both inputs because each local derivative is 1. A multiplication sends the incoming gradient multiplied by the other input.

The chain rule multiplies the effect along a path: ∂L/∂a = (∂L/∂d)(∂d/∂(a*b))(∂(a*b)/∂a) = (−2)*1*(−3) = 6. Similarly ∂L/∂b = −4. Small finite differences provide a useful check. At a differentiable point with a nonzero gradient, a sufficiently small step in the gradient direction increases the output locally; minimizing a loss calls for moving in the opposite direction.

The intermediate gradients make the calculation inspectable: d.grad = −2 and f.grad = 4; the addition routes −2 to both c and a*b; the multiplication then produces a.grad = 6 and b.grad = −4. Karpathy checks these with small input changes before taking a small step in the gradient direction and observing that L rises.

A neuron combines inputs and weights as n = w1*x1 + w2*x2 + b, then applies an activation function. The bias b shifts the response; the nonlinear activation lets networks express more than a chain of linear transformations. The lecture uses tanh, which smoothly maps real inputs into the range between −1 and 1. If its output is t, the local derivative is 1 − t².

In the example, x1 = 2, x2 = 0, w1 = −3, and w2 = 1. A bias near 6.8814 makes n about 0.8814 and tanh(n) about 0.7071. Seed the output gradient with 1: the local tanh derivative is 1 − 0.7071², approximately 0.5. Both additions pass that 0.5 back, so the weight gradients are 0.5*x1 = 1 and 0.5*x2 = 0. The input gradients are 0.5*w1 = −1.5 and 0.5*w2 = 0.5.

The same backward mechanism works whether tanh is implemented as one operation with its own local derivative or decomposed into exponentials and arithmetic. For the lecture's example, an input of zero makes the gradient of its corresponding weight zero for that calculation: changing that weight cannot affect the product while the input is zero.

Each resulting Value stores a backward function that propagates its incoming gradient through its own operation. To automate the order, visit the graph in topological order—parents before their results—then process that list in reverse. By the time a node propagates its gradient, every downstream contribution must have arrived. The output's gradient is seeded with 1.

The following is schematic pseudocode, distilled from the notebook rather than a complete runnable implementation. Each closure remembers the operands and result created during the forward pass; out.grad is the upstream derivative of the final output with respect to out:

add(a, b):
    out = Value(a.data + b.data, parents={a, b})
    out._backward = () => { a.grad += out.grad; b.grad += out.grad }
    return out
multiply(a, b):
    out = Value(a.data * b.data, parents={a, b})
    out._backward = () => {
        a.grad += b.data * out.grad
        b.grad += a.data * out.grad
    }
    return out

For L = (a*b+c)*f, the forward pass records multiplication, addition, then multiplication. The reverse pass seeds L.grad = 1, sends −2 to d and 4 to f, sends −2 to each input of the addition, and finally multiplies that upstream −2 by the other operand to reach a.grad = 6 and b.grad = −4.

The traversal can likewise be written as pseudocode. It visits each distinct node once, appends it only after its parents, and executes the resulting list backwards:

visit(node):
    if node not in visited:
        add node to visited
        for parent in node.parents: visit(parent)
        append node to topo
visit(output)
output.grad = 1
for node in reverse(topo): node._backward()

85:00View full image

A subtle bug appears if the backward function assigns a gradient rather than adding to it. For b = a + a, both paths contribute 1, so db/da = 2; assigning twice leaves 1. The visited set and graph picture contain a only once, but the addition closure still has two operand references and executes both contributions; the picture does not visibly distinguish both edges. Branches that later rejoin have the same problem. Use += for each contribution and start gradients at zero. Ordinary numbers can be wrapped as Values; reverse operators, constant powers, subtraction and division extend the set of expressions. The power rule shown assumes a constant exponent.

The decomposition also shows why the abstraction boundary is flexible. tanh(x) = (exp(2x) − 1)/(exp(2x) + 1) can be one operation or a graph of smaller ones. The lecture implements both and checks that the output and leaf gradients agree. It adds exponentiation and the constant power rule d(x^k)/dx = k*x^(k−1); division becomes multiplication by a reciprocal, and subtraction becomes addition of a negation.

A Neuron owns weights and a bias; a Layer collects neurons; a multilayer perceptron (MLP) passes one layer's outputs to the next. In the example, three inputs feed two hidden layers of four neurons each and one output neuron, for 41 trainable weights and biases. Inputs are treated as fixed data; these parameters are what training changes.

110:23View full image

The forward construction is straightforward: each neuron computes tanh(sum(w_i*x_i) + b); a layer evaluates all its neurons on the same input, and the MLP passes each layer's outputs to the next. Parameters are collected by gathering each neuron's weights and bias across all layers.

The notebook's four input–target pairs are:

[2.0, 3.0, -1.0]  ->  1.0
[3.0, -1.0, 0.5]  -> -1.0
[0.5, 1.0, 1.0]   -> -1.0
[1.0, 1.0, -1.0]  ->  1.0

111:23View full image

For predictions p1 through p4, the displayed loss is (p1−1)² + (p2+1)² + (p3+1)² + (p4−1)². A loss combines the squared differences between predictions and targets into one number. The lecture calls this mean squared error, although the displayed exercise sums the squared terms. Lower loss means the predictions fit these training examples more closely. Because all four forward passes share parameters, their contributions accumulate in those parameters during the backward pass.

The training loop evaluates predictions and loss, clears parameter gradients, calls backward on the loss, and updates every parameter: p.data += −learning_rate * p.grad. Then it repeats. The learning rate sets the step size: too small can make progress slow; too large can leave the region where the local gradient is informative and cause instability. In the demonstration a large step briefly destabilizes training before landing at low loss; that lucky outcome is not evidence that the step size is reliable. Those repeated manual updates also retained old parameter gradients, a bug Karpathy identifies later, so the behavior cannot be attributed to the learning rate alone.

In one manual update after reinitializing the network, a weight with a negative loss gradient increases from about 0.854 to 0.857; after updating all parameters and recomputing the forward pass, loss falls from about 4.84 to 4.36. These are observations from that random initialization, not guaranteed values for another run.

124:43View full image

In compact pseudocode, the order is:

predictions = network(inputs)
loss = sum((prediction - target)^2)
for parameter in parameters: parameter.grad = 0
loss.backward()
for parameter in parameters: parameter.data -= learning_rate * parameter.grad

Karpathy catches a second, distinct accumulation problem in his own loop. The += inside a backward pass is necessary to combine paths, but if parameter gradients are not reset before the next backward pass, old iterations are added too. His initial loop appears to work on the easy four-example problem despite this bug. After reinitialization, the corrected loop clears every parameter's grad before each backward pass and descends more slowly toward low loss; an apparently improving loss alone does not prove the loop correct.

The lecture compares the scalar calculation with single-element PyTorch tensors configured for gradients and matching precision, then compares forward values and gradients. A tensor is an array of scalars: for example, a 2-by-3 tensor holds six numbers. For a fair comparison with Python's double-precision values, the demonstration casts the one-element tensors to double and explicitly enables gradient tracking on the leaves. Calling backward yields the same example gradients; .item() extracts the number from a one-element tensor. The 2022 repository used ReLU where the classroom implementation used tanh; its tests compare micrograd and PyTorch, and its code may have changed since the recording.

A larger demonstration adds batches—subsets of training examples processed per update—a maximum-margin loss, L2 regularization, and a decreasing learning rate. Karpathy mentions regularization in relation to overfitting and generalization without teaching those topics in detail. He also finds the same tanh backward formula inside PyTorch's CPU and GPU code, surrounded by handling for devices and data types. A custom operation similarly needs a forward computation and a backward rule. Bigger systems and language models add different architectures, losses and optimization details, while retaining the central pattern: compute a task loss, propagate gradients, and update parameters.