How a neural network works

What a neural network is

Take any function you know, like f(x) = 2x + 1. It has a rule, and it has numbers inside the rule. Change the 2 or the 1 and you get a different function from the same shape of rule. A neural network is the same idea with two changes. The rule is built from one repeating pattern: weighted sum, then bend, then repeat. And the numbers inside are not chosen by anyone. They are adjusted, a little at a time, until the function does what you want on a large set of examples.

Inputs and outputs are always lists of numbers. An image is a list of pixel brightnesses. A piece of text is first cut into tokens, and each token is turned into a list of a few thousand numbers before the network ever sees it. The output is a list of numbers too: for a language model, one score per possible next token. Whatever the task looks like from the outside, on the inside it is numbers in, numbers out.

Everything the network does is arithmetic that a person could do with a pencil, given enough time. A large model has too many numbers for a pencil, on the order of billions, but no step is different in kind from the steps you are about to do. By the end of this guide you will have run a whole network yourself and gotten an exact answer out of it.

One neuron

A neuron takes several input numbers and produces one output number. It does that in two steps. First it computes a weighted sum: each input is multiplied by its own weight, the products are added together, and a fixed number called the bias is added on top. Second, the sum goes through a nonlinearity, a simple function that bends the result. The most common nonlinearity, ReLU, gets its own section next. For now it is just this rule: if the sum is negative, output zero; otherwise, output the sum unchanged.

A weight says how much an input matters and in which direction. A weight of 0.4 means the input pushes the output up, a little. A weight of -0.5 means the input pushes it down, somewhat harder. A weight of 0 means the neuron ignores that input entirely. The bias shifts the threshold. With ReLU, the neuron produces a nonzero output only when the weighted sum climbs above zero, so a positive bias makes it easier to switch on and a negative bias makes it harder.

inputs:  x1 = 3,    x2 = -1,   x3 = 2
weights: w1 = 0.4,  w2 = 0.6,  w3 = -0.5
bias:    b = 0.2
sum = 3*0.4 + (-1)*0.6 + 2*(-0.5) + 0.2
    = 1.2 - 0.6 - 1.0 + 0.2
    = -0.2
output = ReLU(-0.2) = 0       (negative, so the neuron is off)
same inputs and weights, bias raised to b = 0.7:
sum = 1.2 - 0.6 - 1.0 + 0.7 = 0.3
output = ReLU(0.3) = 0.3      (now it is on)

Read the example as a small decision. The neuron weighs three pieces of evidence, two in favor and one against, and with the first bias it lands just below its threshold and says nothing. Raise the bias and the same evidence tips it over. Nothing about the neuron is intelligent. It is a weighted vote with a cutoff.

Every neuron in a network, from this toy to the largest language model, is this same computation. What differs is how many inputs it has (thousands, in a large model), which nonlinearity is used, and above all what the weights are.

Why the nonlinearity matters

The bend is the part that looks optional and is not. Without it a neuron is a weighted sum, and a weighted sum of weighted sums is still a weighted sum. Stack a hundred linear layers and the whole thing collapses, algebraically, into a single linear layer. That single layer can only draw straight lines (or flat planes, in more dimensions) through the data, and most things worth predicting are not straight.

layer 1:  h = 2*x + 1
layer 2:  y = 3*h - 4
together: y = 3*(2*x + 1) - 4 = 6*x + 3 - 4 = 6*x - 1
one layer with weight 6 and bias -1 does the same job.
the second layer added nothing the first could not do.

ReLU, the rectified linear unit, is the common choice today, and it is barely a function at all: zero if the input is negative, otherwise the input unchanged. Two straight pieces with a corner at zero. That corner is enough. Sigmoid and tanh, smooth S-shaped curves that squash any input into a fixed range, were the standard choice for decades. They fell out of favor for deep networks because their flat ends make the training signal shrink as it passes back through many layers. Large language models mostly use smooth relatives of ReLU such as GELU and SiLU, which keep the general shape but round off the corner.

Here is why a corner is enough. Each ReLU neuron is a hinge: flat on one side of some point, a ramp on the other, and where that point sits depends on the weights and bias. Add hinges together with different positions and slopes and you get a shape with many bends. With enough hinges, that piecewise shape can hug any curve you like. Stack layers, and the hinges of the second layer bend the already-bent output of the first, which is how a network carves out regions with complicated borders instead of straight edges.

f(x) = ReLU(x) - 2*ReLU(x - 1) + ReLU(x - 2)
x = 0:    0 - 0 + 0 = 0
x = 0.5:  0.5 - 0 + 0 = 0.5
x = 1:    1 - 0 + 0 = 1
x = 1.5:  1.5 - 2*0.5 + 0 = 0.5
x = 2:    2 - 2*1 + 0 = 0
x = 3:    3 - 2*2 + 1 = 0
a triangle that rises from 0 to 1 and falls back to 0.
no weighted sum of x alone can go up and then come back down.

That bump is itself a small network: a hidden layer of three ReLU neurons, each reading x with weight 1 and biases of 0, -1, and -2, and an output neuron with weights 1, -2, and 1. Slide the biases and the bump moves. Scale the output weights and it grows or shrinks. Put enough of them side by side and you can trace any curve, which is the idea behind the theorem in the universal approximation section below.

A layer is a matrix multiplication

A layer is a group of neurons that all read the same inputs, each with its own weights and bias, and produce their outputs side by side. Nothing in one neuron depends on another neuron in the same layer, so they can all be computed at once. Write each neuron's weights as a row, stack the rows, and you have a grid of numbers: a matrix. The layer has one matrix of weights and one list of biases.

A matrix multiplication, in this setting, means exactly one thing: every output is a weighted sum of every input, using the weights in the corresponding row. It is the weighted-sum step from the last section, done for all the neurons in a layer in a single operation. Then the bias list is added, and the nonlinearity is applied to each number separately.

inputs:    x = [1, 2, 3]
neuron A:  weights [0.1, 0.2, 0.3],   bias 0
neuron B:  weights [1.0, 0.0, -1.0],  bias 0.5
weight matrix W, one row per neuron:
  [ 0.1  0.2  0.3 ]
  [ 1.0  0.0 -1.0 ]
A: 0.1*1 + 0.2*2 + 0.3*3 + 0 = 0.1 + 0.4 + 0.9 = 1.4
B: 1.0*1 + 0.0*2 + (-1.0)*3 + 0.5 = 1 - 3 + 0.5 = -1.5
after ReLU:  h = [1.4, 0]

This is why graphics cards matter. A GPU is built to do enormous numbers of small multiply-and-add operations at the same time, which is what drawing a 3D scene requires and also what a matrix multiplication is. A feed-forward layer in GPT-3 turns a list of 12,288 numbers into 49,152 and then back into 12,288. The first of those two steps alone is a weight matrix with about 604 million entries, and 604 million multiply-adds for every token that passes through. There are 96 such layers, plus the attention layers between them. Nearly all the time a language model spends answering you is spent inside matrix multiplications, and a GPU does them many times faster than a general-purpose processor.

A complete forward pass, by hand

Here is the network the rest of this guide uses, and the one the backpropagation guide trains. It has two inputs, x1 and x2. It has one hidden layer with two ReLU neurons, h1 and h2. It has one output neuron, y, with no nonlinearity, so y can be any number, positive or negative. That makes nine adjustable numbers in total: six in the hidden layer (four weights and two biases) and three in the output layer (two weights and one bias).

h1 = ReLU(0.5*x1 + (-1)*x2 + 1)
h2 = ReLU(1*x1 + 0.5*x2 + (-0.5))
y  = 1*h1 + (-1)*h2 + 0.5

Running an input through the network from left to right is called the forward pass. Take x1 = 2 and x2 = 1.

hidden neuron h1:
  sum = 0.5*2 + (-1)*1 + 1 = 1 - 1 + 1 = 1
  h1  = ReLU(1) = 1
hidden neuron h2:
  sum = 1*2 + 0.5*1 + (-0.5) = 2 + 0.5 - 0.5 = 2
  h2  = ReLU(2) = 2
output neuron y:
  y = 1*1 + (-1)*2 + 0.5 = 1 - 2 + 0.5 = -0.5
result: y = -0.5

That is the whole computation. Two weighted sums, two bends, one more weighted sum. In matrix form, the hidden layer is a 2 by 2 weight matrix and a bias list of two numbers, and the output layer is a 1 by 2 matrix and a single bias. Written that way, the same forward pass reads as multiply, add, bend, multiply, add.

hidden weights W1 = [ 0.5  -1.0 ]   biases b1 = [  1.0 ]
                    [ 1.0   0.5 ]               [ -0.5 ]
output weights W2 = [ 1.0  -1.0 ]   bias   b2 = 0.5
input x = [2, 1]
h = ReLU(W1 * x + b1) = ReLU([1, 2]) = [1, 2]
y = W2 * h + b2 = 1*1 + (-1)*2 + 0.5 = -0.5

Change the input and a hidden neuron can switch off. With x1 = 0 and x2 = 2, the first hidden neuron's sum is 0 - 2 + 1 = -1, so ReLU makes h1 = 0. The second neuron's sum is 0 + 1 - 0.5 = 0.5, so h2 = 0.5, and the output is 0 - 0.5 + 0.5 = 0. For that input the network behaves as if h1 did not exist. This is what the corner in ReLU does in practice: different inputs light up different subsets of neurons, and each subset is a different straight-line function. The detail matters again in the backpropagation guide, because a neuron that is off passes no training signal back.

What could these numbers mean? Pick anything. Say x1 is hours a student studied, x2 is hours of sleep they lost, and y is the predicted change in their test score. Then the network just predicted that two hours of study and one hour of lost sleep will lower the score by half a point. That is almost certainly wrong, and it is wrong because the nine weights were made up for this guide. The architecture is fine. The weights are not. With the right weights, the same nine numbers could make a sensible prediction. Finding them is the entire problem of training, and it is what the rest of this guide leads toward.

Depth and representation

A network with more than one hidden layer is deep, and the layers run in sequence: the outputs of one are the inputs of the next. Each hidden layer's outputs, its activations, are a re-description of the input in new terms. The raw input to an image network is pixel brightnesses. The first layer's activations describe the same image as a set of edges and color patches at each location. The next layer describes it as combinations of those, like corners and textures. Several layers on, the activations respond to eyes, wheels, and text, and the final layer maps that description to a label.

That progression, from edges to parts to objects, is well established for image networks. Researchers have looked at what most strongly excites individual units at each depth and seen the same pattern across many architectures. The general principle is that depth lets the network build complex features out of simpler ones, and that a feature built at layer five is available to every layer after it.

Language models work the same way, but on vectors that stand for tokens. The input to the first layer is an embedding, a list of numbers standing for one token with no context. Each layer of the transformer rewrites that vector using both the feed-forward arithmetic from this guide and attention, which lets the vector for one token take in information from the others. By the later layers, the vector at each position is a contextual representation: the same word gets a different vector in "river bank" than in "bank account", because the layers have folded the context in.

What individual neurons mean is a much less settled question. Researchers have found a minority of units that respond to something recognizable, like a particular language or a specific kind of syntax, but most do not correspond to any single idea. The evidence so far suggests that networks pack more features than they have neurons, so one neuron takes part in many unrelated features and one feature is spread across many neurons. Reading a large model neuron by neuron is an open research problem, not a solved one.

Where the intelligence comes from

Nothing so far explains why a network is good at anything. It is not the architecture. The nine-weight network above and a language model share the same shape of computation, and with random weights the language model produces gibberish just as reliably as the toy produced -0.5. Everything the network can do lives in the specific values of its weights, and those values come from training.

Training works like this. Start with random weights. Run a forward pass on an example whose correct answer is known. Measure how wrong the output was with a loss function, a single number that is zero when the output is right and grows as it gets worse. If the toy network was supposed to output 1 and produced -0.5, the error is 1.5, and a common loss is the square of that, 2.25. Then, for every weight in the network, work out whether nudging it up or down would shrink the loss, and nudge it that way, a small step. Repeat over millions or billions of examples. That nudging is gradient descent, and the method for working out every nudge at once is backpropagation.

Backpropagation is the subject of its own guide, which trains this same nine-weight network step by step so you can see which weight moves and by how much. The broader picture, what data is used, how long it takes, and how a base model becomes a chat assistant, is covered in how AI models are trained. For this guide the point is only this: the intelligence you see is a property of a particular set of weights, arrived at by a search, and the network is the container.

Kinds of networks

The network in this guide is fully connected: every neuron in a layer reads every output of the layer before it. Fully connected layers are the general case, and they appear inside nearly every other kind of network, but they are wasteful for inputs with structure. A photo has a million pixels, and a fully connected first layer would need a million weights per neuron, with nothing to tell it that nearby pixels are related.

• A convolutional network reuses the same small set of weights at every position of the image. One neuron looks at a 3 by 3 patch, and the same nine weights slide across the whole picture. That makes the network far smaller and builds in the fact that an edge is an edge wherever it appears. Convolutional networks were the standard for images from 2012 until transformers caught up.
• A recurrent network handles sequences by reading one element at a time and carrying a summary vector forward. It was the standard for text until about 2017 and has largely been replaced, because it processes tokens one after another and struggles to keep hold of things it read many steps ago.
• A transformer handles sequences with attention: every position computes weighted sums over every other position, with weights that depend on the content. Attention layers alternate with fully connected feed-forward layers, so a transformer is a stack of exactly the arithmetic from this guide, interleaved with a second operation that moves information between positions.

A large language model is a very large network of the kind described here, with attention layers between the feed-forward ones. In the original transformer layout, about two thirds of the weights in each block sit in the feed-forward layers, not in attention. The dividing line between these families and the older, hand-built approaches is drawn in machine learning vs deep learning; the short version is that deep learning means letting a multi-layer network learn its own features instead of being handed them.

What universal approximation means and does not mean

A theorem from 1989 says that a network with a single hidden layer, if that layer is wide enough, can approximate any continuous function as closely as you want. Cybenko proved it for sigmoid neurons, and Hornik, Stinchcombe, and White proved it more generally the same year. The bump from the nonlinearity section is a hint of why: a hidden layer can build a bump anywhere, and enough bumps of the right heights can trace any curve.

The theorem gets quoted more than it deserves, so it is worth being exact about what it says. It says that for any reasonable function, some set of weights exists that approximates it. It says nothing about how many neurons that takes, and for some functions the answer is astronomically many. It says nothing about whether training will find those weights; gradient descent can settle on something worse. And it says nothing about your data. A network that could in principle represent the right function will still learn the wrong one if the examples do not pin it down.

Researchers have also shown that depth changes the picture: there are functions a deep network represents with a modest number of neurons that a single hidden layer would need exponentially many to match. That is a large part of why practical networks are deep rather than merely wide. The theorem tells you the space of functions is big enough. Everything hard is in finding your way through it.

Common misconceptions

Most confusion about neural networks comes from the vocabulary, which was borrowed from biology and from ordinary speech and fits neither well. Four cases come up constantly.

The neurons are not biological neurons

An artificial neuron is a weighted sum with a cutoff. A biological neuron is a cell with thousands of chemical synapses, spiking behavior, timing effects, and internal state of its own. The researchers who coined the term in 1943 were building a deliberately simplified caricature, and the field kept the name. The resemblance is loose, and no one has shown that the brain learns by anything like backpropagation.

It does not store facts as entries

There is no row in the network that says the capital of France is Paris. The fact is spread across weights in many layers, in a form no one can read off directly, and it is retrieved by the same arithmetic that produces everything else. That is why a model can state a fact correctly in one phrasing and get it wrong in another, and why removing one specific fact from a trained model is a hard, unsolved problem.

The output is deterministic

Given the same weights and the same input, the forward pass produces the same numbers. A language model's final layer produces a score for every possible next token, and the variety you see between two runs of the same prompt comes from the step after the network: the next token is drawn at random from those scores, weighted by how high they are. Turn that sampling off and pick the top score every time, and the same prompt gives the same answer. (In practice, floating-point arithmetic on a GPU can differ in its last digits from one run to the next, which occasionally flips a close call, but that is a numerical detail, not the source of the variety you see.)

Parameters are not neurons

A "7B" model has seven billion parameters, meaning weights and biases, not seven billion neurons. Neurons are far fewer, because every neuron in a fully connected layer carries thousands of weights. Llama 2 7B, for instance, has 32 feed-forward layers with 11,008 neurons each, about 350,000 in all, next to its seven billion weights. When a model is described by its size, the number is always the parameter count; model parameters goes into what that count includes.