By the end you will be able to follow a token from the input text through attention, the feed-forward block, and the residual stream to the probability of the next token.
A transformer is a stack of identical layers. Each layer does two things to every token in the input: it lets the token gather information from the tokens before it (attention), and it then processes what the token has gathered on its own (the feed-forward block). Repeat that a few dozen times and read a probability distribution off the last token. That is the whole model.
This guide walks through the mechanism at the level of the actual vectors and arithmetic, with a small worked example for attention. It assumes you have read how transformers work and want the real mechanism. It does not assume linear algebra: where a dot product or a matrix appears, one sentence says what it does.
Before 2017, the standard way to process text with a neural network was a recurrent network. It read one token at a time, left to right, and kept a single vector as its memory of everything it had seen so far. That vector had a fixed size, so the network had to compress a whole paragraph into a few hundred numbers and keep compressing as more text arrived.
Two problems followed from that design. First, information about distant words faded. By the time the network reached the verb at the end of a long German sentence, its memory of the subject at the start had been overwritten dozens of times. Second, the computation was sequential by construction. Token 50 could not be processed until token 49 was done, so training could not be spread across the positions of a sequence, and graphics cards sat mostly idle.
Attention had already been bolted onto recurrent translation models by 2015, as a way for the decoder to look back at the source sentence while writing each output word. The 2017 paper by Vaswani and seven colleagues at Google, "Attention Is All You Need", asked what happened if you removed the recurrence and kept only the attention. The answer was a model that trained faster, translated better, and could connect any two positions in a sequence in one step, however far apart they were.
The original was an encoder-decoder: one stack read the source sentence, and a second stack wrote the translation while attending to the first. The GPT line of models, and nearly every large language model since, kept only the second stack and trained it to predict the next token of ordinary text. The rest of this guide describes that decoder-only model.
The model never sees letters. Text is first cut into tokens, pieces that are usually a word or part of a word, and each token is replaced by an integer id from a fixed vocabulary of tens of thousands of entries.
The first layer of the model is a lookup table. Each vocabulary id has a row in it, and each row is a vector: a list of a few thousand numbers (512 in the 2017 paper, 4,096 in a typical 8-billion-parameter model today). Those vectors are learned during training along with everything else, and the embeddings guide covers what ends up encoded in them. For this guide the important fact is simpler: after the lookup, the input is a list of vectors, one per token, all the same width, and every layer from here on reads and writes vectors of that width.
Attention, as the next section shows, computes the same thing regardless of the order of the tokens. If you shuffled the input vectors, each token's attention output would be identical; only the list order would change. Since dog bites man and man bites dog should not mean the same thing, position has to be injected on purpose.
The 2017 paper did this by adding a fixed pattern of sine and cosine waves of different frequencies to each embedding, so that position 7 got a slightly different vector added to it than position 8. It has largely been replaced.
Most current models use rotary position embeddings, or RoPE. Instead of adding a position signal to the token vector, RoPE rotates the query and key vectors inside attention by an angle that grows with the token's position. Comparing a vector rotated by angle a with one rotated by angle b depends only on the difference between a and b, so the attention score between two tokens ends up depending on how far apart they are, not on where in the text they sit. Llama, DeepSeek, Kimi, and Qwen all use a version of it.
Attention is the step where a token gathers information from other tokens. Everything in it is ordinary arithmetic on vectors; the names are the confusing part.
The model multiplies each token's vector by three learned matrices to produce three new vectors. (A matrix here is a grid of numbers. Multiplying a vector by it produces a new vector in which each output number is a weighted mix of the inputs, with weights learned in training.)
it, a query might, loosely, encode "I need a noun that could be the thing I refer to."trophy, a key might encode "I am a singular concrete noun."Those glosses are for intuition. The matrices are learned from data, so a real query is not a sentence but a list of numbers that happens to match certain keys.
The score between a query and a key is their dot product: multiply the first number of one by the first number of the other, the second by the second, and so on, then add the products. A dot product is large when the two vectors point in the same direction and have some length, near zero when they are unrelated, and negative when they point opposite ways. It measures how well a key matches a query.
Each score is then divided by the square root of the key vector's width. With a width of 64 (the paper's choice) that is 8. Without this step, wider vectors produce bigger dot products just because there are more terms in the sum, and the next step becomes lopsided.
The next step is the softmax. It takes the list of scaled scores, raises e (about 2.718) to the power of each one, and divides each result by the sum of all of them. The outputs are positive and add up to exactly 1, so they can be read as weights. Softmax also exaggerates differences: a score two units above another gets about seven times the weight.
Finally, the token's output is the weighted sum of the value vectors: each value multiplied by its weight, all added together. That output is what attention hands to the rest of the layer.
tokens: The, cat, sat. query comes from "sat"
q(sat) = [1, 2]
keys: k(The) = [-1, 1] k(cat) = [1, 1] k(sat) = [2, 0]
values: v(The) = [0, 2] v(cat) = [2, 1] v(sat) = [1, 0]
dot products of q with each key:
The: 1*(-1) + 2*1 = 1
cat: 1*1 + 2*1 = 3
sat: 1*2 + 2*0 = 2
divide by sqrt(2) = 1.41:
The: 0.71 cat: 2.12 sat: 1.41
softmax:
e^0.71 = 2.03 e^2.12 = 8.33 e^1.41 = 4.10 sum = 14.46
weights: The 0.14 cat 0.58 sat 0.28 (total 1.00)
output = 0.14*[0,2] + 0.58*[2,1] + 0.28*[1,0]
= [0, 0.28] + [1.16, 0.58] + [0.28, 0]
= [1.44, 0.86]
In this toy, sat pulls most of its new information from cat, some from itself, and a little from The. In a real model the vectors are 128 numbers wide per head rather than 2, and there are thousands of tokens rather than 3, but the sequence of operations is exactly this one.
In a decoder-only model, a token is only allowed to attend to itself and the tokens before it. This is done by setting the score for every later position to negative infinity before the softmax, which makes its weight exactly 0. The token at position 5 sees positions 1 through 5 and nothing else.
The reason is training. The model learns by predicting the next token at every position of a text at once, and if position 5 could see position 6, that prediction would be trivial. The mask makes every position an honest prediction problem, and it is also what makes generation possible: a model that never looked ahead during training can produce text one token at a time without noticing the difference.
The description so far is one attention head. Real layers run several in parallel. The token vector is split into slices (the 2017 paper used 8 heads of 64 numbers each on a 512-wide vector; a 4,096-wide model today might use 32 heads of 128), each slice gets its own query, key, and value matrices, and each head runs the whole score-softmax-sum routine on its own slice. The outputs are concatenated back into one full-width vector and multiplied by one more learned matrix that mixes them.
The point of heads is that one weighting cannot serve every purpose. A token may need to find its subject, its nearest adjective, and an earlier mention of itself, and those are three different match patterns.
What individual heads learn is an open research question. In small models, researchers have found heads with clean, describable jobs. Some attend almost entirely to the previous token. Others, called induction heads, do something more useful: when the current token is X, they look for an earlier X in the context, attend to whatever came right after it, and copy that forward. If the text earlier contained Harry Potter and the current token is Harry, an induction head attends to the earlier Potter and pushes the model to predict it. Olsson and colleagues (2022) showed that these heads appear abruptly during training in small models, and that their appearance coincides with a jump in the model's ability to use examples given in the prompt. The paper argues that induction heads are a large part of what in-context learning is, while calling its own evidence for large models indirect. The evidence so far supports a partial account, not a complete one.
Attention does not rewrite the token vectors. It produces one new vector per position, which the residual step below adds to the old one, so a token that finds nothing useful in the context can come out nearly unchanged.
After attention, each token's vector goes through a second sub-block that touches no other token. It is a plain two-layer network: multiply by a matrix that expands the vector to about four times its width (512 to 2,048 in the paper; 4,096 to about 14,000 in a typical model today), apply a simple nonlinearity to each number, then multiply by a second matrix that brings it back to the original width. The nonlinearity was originally ReLU, which replaces negative numbers with zero. Most current models use a smoother variant with a gate, but the shape is the same. The neural network guide explains why a nonlinearity is needed at all.
The block runs on every position separately and identically. It only has what attention already gathered into that position. Attention moves information between positions; the feed-forward block processes it in place. That division of labor is the whole layer.
Two facts about this block matter more than its shape. First, it is where most of the parameters are. With the four-times expansion, the two matrices together hold eight times the square of the model width, against four times the square for the four attention matrices, so roughly two thirds of each layer sits here. Second, interpretability researchers have found evidence that these blocks behave in part like key-value memories: rows of the first matrix act as patterns that fire on particular inputs, and columns of the second hold what to add when they fire. Geva and colleagues (2021) showed this in a small model, with early layers keying on surface patterns and later layers on meaning. That reading is partial, but it is the best current picture of where a model keeps facts.
In mixture-of-experts models this is the block that gets multiplied. Instead of one feed-forward network per layer there are many (DeepSeek-V3 has 256 routed experts per layer and picks 8 of them for each token), and a small router chooses which ones run. Only the chosen experts do any work, so the model can hold far more parameters than it uses on any single token. Attention is not split this way; every token still goes through the same attention.
Neither sub-block replaces the token's vector. Each one computes an update and adds it to what came in. Written out, the attention step is x = x + attention(x) and the feed-forward step is x = x + feedforward(x). The addition is the residual connection.
The consequence is that one vector per position runs straight through the whole model, from the embedding lookup to the final output, with each sub-block reading from it and adding something back. Interpretability researchers call this the residual stream, and the name is a good mental model. The layers are not a pipeline; they are contributors writing into a shared channel. A layer can leave the stream alone, add a small correction, or add something that a much later layer will read.
Residual connections are also the main reason a stack of a hundred layers can be trained. Training works by backpropagation, which sends an error signal backward through every layer to tell each parameter how to change. Through a plain stack of a hundred transformations that signal shrinks or explodes, and the early layers stop learning. Through a residual stack the signal has a direct path (the plain x in x + f(x)) that passes it back unchanged, so every layer gets a usable signal however deep it sits.
Normalization does the other half of the job. Before each sub-block, the token's vector is rescaled so that its numbers have a standard spread, then multiplied by a learned gain. (The original layer normalization also centers the numbers; the simpler RMSNorm used in most current models only rescales.) Without it, the residual stream grows as layers add to it, and deep sub-blocks would receive inputs far larger than shallow ones, which makes training unstable. The 2017 paper normalized after each addition. Nearly every model since normalizes before each sub-block, which trains more reliably at large scale.
One transformer layer is now fully described.
There is nothing else in a layer. The whole model is this layer repeated with fresh parameters: 6 times in the 2017 decoder, 12 to 48 in the GPT-2 family, 96 in GPT-3, 126 in the largest Llama 3, 61 in DeepSeek-V3 (with each layer growing far wider as well). Layers do not share weights; layer 40 has its own matrices, learned independently of layer 39's.
At the top of the stack, each position holds a vector that every layer has read from and written to. To predict the next token, the model takes the vector at the last position, normalizes it one final time, and multiplies it by the unembedding matrix: one row per vocabulary entry, each row a vector of the model's width. The result is one score per vocabulary token, called a logit.
A softmax over those scores (the same operation as in attention, now over a hundred thousand entries instead of three) turns them into probabilities. One token is drawn from that distribution. The drawn token is appended to the input, and the whole stack runs again to produce the token after it. That loop, one forward pass per token, is what generation is.
Temperature is a knob on that final softmax. Every logit is divided by the temperature before the softmax runs. Near zero, the highest score dominates and the model almost always picks its most likely token, so the output becomes repetitive. At 1, the draw uses the model's raw distribution. Higher values flatten it, so unlikely tokens come up more often and the text gets more varied and, past a point, incoherent. Temperature changes nothing about what the model computed, only how the final draw is made.
During training, the model gets a whole document at once and is asked, at every position simultaneously, to predict the token that follows. Because the causal mask already stops each position from seeing its future, all positions can be computed in one pass: one large matrix multiplication for the queries of every token, one for the keys, one for the values, and so on up the stack. A sequence of 8,000 tokens yields 8,000 prediction problems from a single forward pass. This is what recurrent networks could not do, and it is why the transformer took over. Graphics processors are built for exactly this kind of large multiplication, and the architecture hands them almost nothing else.
Generation runs the other way. Each new token depends on the one before, so tokens come out one at a time. Done naively, producing token 1,000 would mean recomputing the keys and values of the previous 999 tokens at every layer, which the model already computed when it produced them. So they are stored. The KV cache keeps every earlier token's key and value vectors at every layer. A new token computes only its own query, key, and value, scores its query against the cached keys, and appends its own key and value to the cache.
That cache is where the memory of a long conversation goes, and its size is easy to estimate.
model: 32 layers, 8 key-value heads, 128 numbers per head per layer: key + value = 2 * 8 * 128 = 2,048 numbers per token: 32 layers * 2,048 = 65,536 numbers at 2 bytes per number: 131,072 bytes = 128 KB per token a 100,000-token context: about 13 GB of cache
Those numbers are the shape of a typical 8-billion-parameter open model, and the 13 gigabytes come on top of the roughly 16 gigabytes the model's own weights take at the same precision. The eight key-value heads, rather than one per query head, are a common economy called grouped-query attention, adopted mainly to shrink this cache. DeepSeek's models go further and store one compressed vector per token from which the keys and values are rebuilt.
The cache is also why the context window is a hard limit rather than a soft one. Three things run out. Memory, because the cache grows with every token. Time, because each new token scores its query against every cached key, so the work per token grows with the length of the context and the total work for a conversation grows with its square. And position, because the model was trained on sequences up to some length, and its position encoding has never seen anything past it. Stretching RoPE to longer contexts is possible and common, but it is a separate engineering step.
A parameter is one learned number, and the count is what the 8B or 70B in a model name refers to. The architecture above says where they live, and a rough tally shows the proportions.
width d = 4,096 layers = 32 vocabulary = 128,000 per layer: attention: 4 matrices of d*d = 4 * 16.8M = 67M feed-forward: 2 matrices of d*4d = 2 * 67.1M = 134M layer total: 201M 32 layers: 32 * 201M = 6.4B embedding table: 128,000 * 4,096 = 0.5B unembedding matrix: another 0.5B total: about 7.5B
Real models adjust the shapes (shared key-value heads shrink attention, and gated feed-forward blocks use three matrices instead of two), which is how a model of this shape lands near 8 billion in practice, but the proportions hold. About two thirds of the layer parameters are feed-forward, one third are attention, and the embedding tables are a slice that shrinks as models grow. In a mixture-of-experts model the feed-forward share is larger still, since every expert is a full feed-forward block and the attention matrices are not multiplied.
Everything above applies to Llama, DeepSeek, Kimi, and Qwen, whose weights and configurations are public, and, as far as their makers have said, to GPT and Claude as well. The differences between those models, in what they know and how they behave, do not come from the architecture. They come from three other places: how many parameters the model has and how they are divided among layers and experts, what text it was trained on and how much, and what happened after pre-training, the post-training that turns a next-token predictor into an assistant with a particular manner. Two models with the same architecture and the same size can behave very differently.
Nor does the architecture tell you what the model understands. It tells you the computation: lookups, dot products, softmaxes, weighted sums, and additions, repeated. Whether the vectors that flow through that computation amount to meaning, or a compressed record of the training data, or something without a good name yet, is a live argument that the mechanism alone does not settle. What the mechanism does settle is narrower. There is no stored text being searched, and there is no step that resembles a person reading. There is a specific, inspectable, and still only partly understood computation.
Every model Berges AI serves (Kimi, DeepSeek, GLM, and Qwen) is a decoder-only transformer of the kind described here, and all four are mixture-of-experts variants, so the feed-forward block in each of their layers is split into experts. The model you pick from the sidebar is a choice between different sizes and training runs of the same architecture, all of them reasoning models. The Berges layer that strips flattery and padding sits outside the model entirely, in the instructions each message carries.
Try Berges AIAn encoder lets every token attend to every other token in both directions, which suits tasks that read a whole input at once, such as classification or producing embeddings. A decoder masks attention so each token sees only earlier ones, which is what lets it generate text one token at a time. The 2017 model used both. Nearly all large language models today are decoder-only.
The name predates the transformer. Around 2014, researchers added a mechanism to translation models that let the decoder focus on different words of the source sentence as it wrote each output word, and called it attention because the weights showed where the model was looking. The transformer kept the name and the weighting and dropped everything else.
Only loosely, and the word should not be read as a claim about neuroscience. Both involve weighting some inputs more than others, and that is where the resemblance ends. Transformer attention is a fixed formula of dot products and a softmax with no known biological counterpart, and artificial neurons do not work like biological ones.
It is the stored key and value vectors of every earlier token at every layer, kept in memory during generation so that each new token only has to compute its own. Without it, producing each token would mean redoing the work for the whole context. It is the main memory cost of a long conversation, and it grows with every token.
Yes. Vision transformers cut an image into patches and treat the patches as tokens, and the same design is used for audio, protein sequences, and the mixed text-and-image models now common. Attention turned out to be a general way to relate the parts of a sequence, not something specific to language.
Three reasons. The KV cache grows with every token, so memory runs out. Each new token attends to every earlier one, so the work per token grows with the context and the total work grows with its square. And the model was trained on sequences up to a fixed length, so its position encoding has not seen longer ones, and extending it takes deliberate extra training or adjustment.