What a neural network is: from pixels to a parameterized function

A handwritten 3 can be recognized even when its pixels differ substantially from those of another 3. Writing explicit rules that handle all those variations is much harder. The video builds a mathematical object that can learn the task: a layered network whose behavior is controlled by numerical parameters. This chapter explains its structure; how the parameters are learned is reserved for the next chapter.

The opening describes a program returning a number “between 0 and 10.” In the worked network, the ten outputs are explicitly labeled 0 through 9. The former wording is therefore left as spoken; the output interpretation below follows the displayed example. Watch the opening at 00:49.

Think of a neuron in this example as holding an activation, a number indicating how strongly it is lit. A 28×28 grayscale image gives 784 input neurons, one per pixel: black is 0 and white is 1. The final layer has ten neurons, one for each digit from 0 to 9; their activations express how much the network associates the image with each digit. The highest output is its choice.

Between them are two hidden layers, each with 16 neurons. The presenter chose two layers to motivate the explanation and 16 because they fit on screen; these are illustrative choices, not a recipe for every network. Here activations are between 0 and 1, but that range is also specific to the example. The brain is a loose inspiration for connected neurons, not an assertion that the diagram reproduces biology.

04:16View full image

The illustrated architecture has 784 input neurons, two hidden layers of 16, and ten digit outputs.

In the already trained example, pixel brightness produces a pattern in the first hidden layer, which produces a pattern in the next, then ten output activations. Understanding exactly how one pattern determines the next is the central question. Watch at 02:43.

The motivation for layers is that a complicated pattern might be assembled from simpler ones. A 9 has an upper loop and a line; an 8 has two loops. One might hope that a neuron in the second hidden layer responds to an upper loop in many different shapes, and that the output layer combines such components to distinguish digits.

That only moves the question back a step: how could the network recognize a loop? The proposed intuition is that smaller edges could combine into loops or longer strokes. A 9 might activate several edge detectors, then detectors for an upper loop and a vertical line, and finally its digit output. Similar levels of abstraction can be imagined for speech: sounds form syllables, words, phrases and more abstract thoughts.

07:22View full image

The video illustrates a possible decomposition of a loop and a stroke into smaller edges.

This is a hope about the hidden layers, not an observation that this trained network actually works that way. The presenter explicitly leaves that empirical question until after training is discussed. Watch at 05:32.

To make the hope concrete, consider one neuron that might respond to a short bright edge in a particular region. Each connection from an input pixel has a weight, a number multiplying that pixel's activation. Add all these products to get the weighted sum: w1 × a1 + w2 × a2 + ... + wn × an. Here each a is an input activation and each w is its associated weight.

Set most weights to zero and give the desired region positive weights: brighter pixels there increase the sum. Give surrounding pixels negative weights: brightness around the desired line reduces it. The sum is high when the center is bright and its surroundings are dark. The colors in the animation distinguish positive green weights from negative red ones; the numerical values, not labels such as “edge,” determine the computation.

09:55View full image

The weighted sum combines pixel activations through positive and negative connections; the highlighted region illustrates the desired edge.

This is a designed example of what adjustable weights could express. It does not establish that a particular trained neuron detects that edge. Watch at 08:21.

A weighted sum can be any real number, while this example's activation is between 0 and 1. The sigmoid function, also called a logistic curve, maps very negative inputs close to 0 and very positive inputs close to 1, increasing smoothly between them.

A bias is another adjustable number added before sigmoid. For example, adding −10 shifts the response so a weighted sum around 10 reaches sigmoid's midpoint: stronger evidence is needed for a high activation. It is a smooth transition, not a hard on/off switch. In shorthand, the neuron's activation is sigmoid(weighted sum + bias). Weights shape which pattern contributes; bias shifts how much evidence is needed.

11:35View full image

Adding a bias of −10 shifts the sigmoid response toward a weighted sum of 10.

Every one of the first hidden layer's 16 neurons connects to all 784 inputs. That transition alone has 784 × 16 weights plus 16 biases, one bias per receiving neuron. The later transitions have their own weights and biases, bringing the chosen network to almost 13,000 parameters in all.

Learning, in the sense introduced here, means getting the computer to find settings for those numbers that solve the task. The video does not yet explain the procedure. Imagining setting them by hand can nevertheless help us reason about the machinery. If a network behaves unexpectedly, examining weights and biases can suggest experiments; if it works for unexpected reasons, investigation can challenge our assumptions about what it learned.

11:59View full image

The first hidden layer alone has 784 weights per neuron and one bias per neuron.

Writing a separate weighted sum for every neuron quickly becomes cumbersome. Put the previous layer's activations in a column vector, and arrange the connection weights in a matrix: each row contains the weights feeding one neuron in the next layer. Multiplying the matrix by the activation vector calculates all those weighted sums together.

Put the receiving neurons' biases in another vector, add it to the product, then apply sigmoid separately to each resulting component. In compact notation, next activations = sigmoid(Wa + b). The expression describes the same individual operations, packaged as a layer transition. Matrix operations also allow code to use optimized numerical libraries.

14:55View full image

The matrix expression collects the weighted sums and biases for a layer before applying sigmoid to each result.

A neuron is more than a static number: in this example it is a function of the preceding layer's activations. Repeat the layer calculation and the entire network becomes a function from 784 input numbers to ten output numbers. Its roughly 13,000 weights and biases specify which function the example computes.

15:40View full image

The same network can be viewed as a composition of functions, from image pixels to output activations.

The remaining question is how data can lead to useful parameter settings, and what this particular trained network actually does internally. The presenter reserves both for the next video. Watch at 15:17.

At the end, the presenter speaks with Lisha Lee, introduced as having done doctoral work on deep-learning theory and then working at Amplify Partners. The presenter says the firm provided some funding for the video. They discuss sigmoid's use in early networks and a loose biological analogy of neurons being active or inactive.

In that 2017 conversation, they describe sigmoid as harder to train in some deep networks and ReLU as having worked well for very deep networks. ReLU means rectified linear unit: ReLU(a) = max(0, a). A negative input becomes zero; a positive input passes through unchanged. Lee describes the active/inactive analogy as a simplification. This historical discussion is not a claim that sigmoid has no present uses, nor that all neural activations are bounded by 1. Watch at 17:03.