By the end you will know how a token becomes a vector, what "close" means between two vectors, how the numbers are learned, and what similarity can and cannot tell you.
A language model splits text into tokens, and then the first thing it does with each token is throw the text away. What it keeps is a list of numbers, a few hundred to several thousand of them per token. That list is the token's embedding. Every later step in the model works on those numbers and never on the letters.
This raises an obvious question. How can a list of numbers mean anything? The short answer is that the numbers are not chosen. They are learned, and they are learned under one pressure: tokens that show up in the same kinds of places should end up with similar numbers. Everything useful about embeddings follows from that pressure, and so do most of their failures.
A neural network is arithmetic. It multiplies inputs by weights, adds the results, passes them through a simple function, and repeats. It cannot take the string "cat" as an input. Something has to turn each token into numbers first, and the way that is done matters more than it looks.
The obvious move is to number the vocabulary: cat is 1, dog is 2, car is 3, truck is 4. The trouble is that the network treats a number as a quantity. It will compute that truck is four times cat, that dog is halfway between cat and car, and that cat plus dog equals car. None of that is true of the words. A plain numbering smuggles in structure that is not there, and the network has to spend effort unlearning it.
The classic fix is the one-hot vector. Give every word its own slot in a long list, put a 1 in that slot and a 0 in all the others. Now no word is a multiple of another, and no word is the average of two others.
slots: cat dog car truck cat = [ 1, 0, 0, 0 ] dog = [ 0, 1, 0, 0 ] car = [ 0, 0, 1, 0 ] truck = [ 0, 0, 0, 1 ] cat . dog = 1*0 + 0*1 + 0*0 + 0*0 = 0 cat . truck = 1*0 + 0*0 + 0*0 + 0*1 = 0 every pair of different words scores 0
One-hot removes the false structure by removing all structure. Every word is exactly as far from every other word: cat is no closer to dog than it is to truck. With a vocabulary of 50,000 tokens the list is 50,000 slots long, 49,999 of them zero, and the network has to learn everything about every token separately, with no hint that two of them might behave alike. An embedding keeps what one-hot got right, that no token is a quantity, and puts back the thing it threw away, which is similarity.
An embedding gives each token a short, dense list of numbers instead of a long list of zeros. Word2vec typically used a few hundred per word. A large language model uses more: GPT-3 used 12,288 per token, and current open-weights models mostly use a few thousand. The list is a vector, each number in it is one dimension, and the full set of vectors, one row per token, is the embedding matrix.
The numbers start random. Training then adjusts them under one rule: tokens that appear in similar contexts should end up with similar vectors. The rule is called the distributional hypothesis, and the linguist J. R. Firth gave it a slogan in 1957: "You shall know a word by the company it keeps." Cat and dog both turn up after the, before food, and near vet, fur, and leash. Truck does not. So cat and dog get pulled together, and truck is left somewhere else.
"Similar vectors" needs an actual measure, and two are in common use. The dot product multiplies the two vectors slot by slot and adds up the results, so it is large when both vectors have big numbers of the same sign in the same slots. Cosine similarity is the dot product divided by the lengths of the two vectors, which cancels out how big each vector is and keeps only how closely they point the same way. It runs from 1 (same direction) through 0 (no relation) to -1 (opposite). When a similarity score is quoted without explanation, it is almost always a cosine.
The dot product rewards long vectors. A token whose numbers are all large scores high against everything, related or not. Cosine similarity ignores length, which is usually what you want when the question is whether two tokens mean similar things.
Real vectors are too long to look at, so here is an invented one: three dimensions, four words. The numbers were chosen so that the two animals point roughly one way and the two vehicles roughly another. The numbers are fake. The arithmetic is exactly what a real model does.
cat = [ 2, 1, 2] length = sqrt(4 + 1 + 4) = 3 dog = [ 4, 4, 2] length = sqrt(16 + 16 + 4) = 6 car = [-2, 1, 2] length = sqrt(4 + 1 + 4) = 3 truck = [-4, 4, 2] length = sqrt(16 + 16 + 4) = 6 cat . dog = 2*4 + 1*4 + 2*2 = 8 + 4 + 4 = 16 cosine = 16 / (3 * 6) = 0.89 cat . car = 2*(-2) + 1*1 + 2*2 = -4 + 1 + 4 = 1 cosine = 1 / (3 * 3) = 0.11 car . truck = (-2)*(-4) + 1*4 + 2*2 = 8 + 4 + 4 = 16 cosine = 16 / (3 * 6) = 0.89
Read off the results. Cat against dog scores 0.89. Car against truck scores 0.89. Cat against car scores 0.11. The animals point almost the same way, the vehicles point almost the same way, and an animal and a vehicle barely agree. That is what "similar meaning" looks like once it is numbers.
Two details of the toy carry over to real models. First, the dot product cares about length. Dog and truck are twice as long as cat and car, so every dot product involving them is inflated. Dog against truck gives a dot product of 4, four times what cat against car gave, but the cosine is 4 divided by 6 times 6, which is 0.11, the same as cat against car. Cosine sees that the two pairs are equally unrelated. The dot product does not. Second, negative numbers are normal. A dimension is not a count of anything. It is a direction, and a token can sit on either side of it.
A real embedding has thousands of dimensions instead of three, and the arithmetic is unchanged: multiply slot by slot, add, divide by the lengths. The reason for thousands is capacity. With three numbers you can separate animals from vehicles. With thousands you can separate animals from vehicles, nouns from verbs, past from present, formal from casual, English from Spanish, and a long tail of distinctions nobody has named, all at the same time and all in the same list.
Nobody writes the vectors by hand, and nobody tells the model that cats resemble dogs. The vectors are a by-product of a prediction task, and that is the single most important fact about them.
The cleanest version is word2vec, published by Tomas Mikolov and colleagues at Google in 2013. Take a large body of text and slide a window across it. At every position, train a very small network to do one of two jobs: guess the middle word from the words around it, or guess the surrounding words from the middle word. The network has one hidden layer of a few hundred units, and its input weights form a table with one row per vocabulary word. When training ends, you throw the network away and keep the table. The rows are the embeddings.
Why does predicting words produce similarity? Because the network has to guess food after the cat wants and also after the dog wants. The cheapest way to be right both times is to give cat and dog rows that look alike, so the layer above can treat them the same. Every shared context nudges the two rows closer. Every context where one word appears and the other never does nudges them apart. Over billions of words the nudges add up to a geometry in which similar use means similar vectors. The model never sees a definition of cat. Similarity falls out of the pressure to predict.
GloVe, from Stanford in 2014, reaches a similar geometry by a different road. It counts how often every pair of words appears near each other across the whole corpus, then fits vectors so that the dot product of two words tracks how often they appear together (the logarithm of the count, to be exact). It is worth knowing about because it makes Firth's slogan literal: the only input is a table of who keeps company with whom.
A large language model has no separate embedding step. Its embedding matrix is the first layer of the network: one row per token in the vocabulary, and looking up a token means selecting its row. The rows start random like every other weight and are adjusted by backpropagation together with all the layers above them, while the whole model learns to predict the next token. The pressure is the same as in word2vec. The difference is that the network above the table is enormous, so each row is shaped by everything the rest of the model needs from it, not only by which words sit nearby.
That is the entire trick. If you know how a neural network learns, you already know how embeddings are learned. They are weights. Nothing about them is special except that they come first.
The result that made embeddings famous came out of the word2vec work. Take the vector for king, subtract the vector for man, add the vector for woman, and find the vocabulary word nearest to the result. You get queen. Paris minus France plus Italy lands near Rome. Walking minus walk plus swim lands near swimming.
What this shows is real. Some directions in the space correspond to consistent relations. The offset from man to woman is roughly the same as the offset from king to queen and from actor to actress, so adding it to any of the first words moves you to roughly the second. Nobody built that in. The pairs are used in parallel ways across the training text, so training pushed them into parallel positions.
man = [1, 2] woman = [1, 5] woman - man = [0, 3] king = [6, 2] queen = [6, 5] queen - king = [0, 3] king - man + woman = [6, 2] - [1, 2] + [1, 5] = [6, 5] which is exactly queen. Real vectors only land near it.
Now the part the demo leaves out. The famous examples are a curated set. On the analogy test sets built for word2vec, the original models got roughly half to two thirds of the questions right, and those sets were made of relations already known to work. Many relations that seem just as regular to a person fail outright. The search for the nearest word also excludes the three input words by convention, and that convention does a lot of work: without it, king minus man plus woman frequently returns king, because the result is still closest to where it started. The arithmetic is a real property of the space. The tidy demo is a selection from it.
It is tempting to imagine that dimension 1 means "animal" and dimension 2 means "vehicle", because that is how the toy above was drawn. Real embeddings do not work that way. Training never assigns a meaning to a slot. It only pushes whole vectors together and apart. So a concept like "royalty" or "plural" ends up as a direction through the space, at some angle to every axis, and reading any single dimension by itself tells you almost nothing.
There is a stronger reason the slots stay unreadable. The evidence so far suggests that models store more concepts than they have dimensions, by using directions that overlap slightly instead of separate axes. If a 4,000-dimensional space holds far more than 4,000 features, then no dimension can be one feature. Every slot is a small part of many things at once.
Interpretability researchers have nonetheless found directions that line up with human concepts. Simple probes trained on the vectors inside a language model can recover directions that track whether a statement is true, where a city is on the map, or when an event happened, and newer techniques pull large numbers of finer-grained features out of the same vectors. That work is young. The directions are found after the fact rather than designed, and how much of the space they account for is an open question. Whether any of it amounts to understanding is the subject of are LLMs stochastic parrots?.
A word2vec vector is fixed. Bank has one row, and that row has to serve river bank and bank loan at once, so it ends up somewhere between the two and close to neither. Every word with more than one sense pays this price, and most common words have more than one sense.
A transformer starts the same way. Its input embedding is one fixed row per token, looked up before the model has seen any context. What changes is everything after the lookup. Each layer of the transformer takes the vector at every position, reads the vectors at the other positions through attention, and writes an updated vector back in place. Layer by layer, the vector for bank absorbs river or absorbs loan. By the middle of the stack, two vectors that began identical have moved far apart, one toward the region of shore and water, the other toward lender and account.
That moving vector is a contextual embedding, and it is what most people mean by "embedding" today. It is not stored in any table. It is computed fresh for every token of every input, and it depends on everything the model has read so far in that input. The fixed table still sits at the bottom, but the geometry that makes the model useful lives in what the layers do to the table's rows.
Token embeddings answer one question: what does this token mean here. Many practical uses need a different one: what is this whole passage about, in a form you can compare against other passages. That takes one vector per text instead of one per token.
The direct route is pooling: run the text through a model, take the contextual vectors of all its tokens, and average them, or take the vector at one chosen position. Out of a plain language model this works worse than you would expect, because those vectors were shaped to predict the next token, not to be compared with each other. Sentence-BERT, from 2019, showed the fix that most embedding models still use. Start from a pretrained transformer, then train it further on pairs of sentences with labeled relationships (same meaning, contradiction, or unrelated), so that the pooled vector is judged on whether it puts similar meanings close. The result is an embedding model. It comes from the same architecture family as a chat model, but its training is different and so is its output: you give it text and get back a vector, not more text.
That one vector per text is behind most of the places embeddings show up in products:
Comparing one query vector against a few thousand stored vectors is a loop. Comparing it against a hundred million is not, and that is what a vector database is for. It stores vectors, builds an index that lets it skip most of them, and answers "which 20 are nearest to this one" in milliseconds, usually trading a small loss of exactness for the speed. It is a search index for vectors. Nothing about it is more mysterious than that.
Train on text from many languages and dog and perro end up near each other without a dictionary. The pressure is the usual one. Perro appears after el, before ladra, and near veterinario and correa, and those words in turn sit near their own contexts, all the way through the Spanish vocabulary. Translated pages and mixed-language text in the training data tie the two vocabularies together, and so do numbers, names, and code, which look the same in every language and act as fixed anchors that each language settles around.
This is part of why a model trained mostly on English can hold a conversation in Spanish. The evidence so far suggests that in the middle layers of a multilingual model, a concept's vector is partly shared across languages, with the specific language showing up more strongly near the input and near the output. A fact the model absorbed from English text can then often be produced in Spanish, because the fact lives in a region of the space that both languages reach. The practical side of this is covered in LLMs in Spanish and other languages.
The alignment is imperfect. Words that translate one to one line up well. Words whose sense boundaries differ between the languages, idioms, and terms with cultural weight line up worse. And a language with little training text gets vectors shaped by little evidence: its geometry is coarser, its rare words drift, and its tokenization is usually worse to begin with, so a single word gets chopped into more pieces with less meaning each. Spanish is well served by today's models. Many languages are not.
Embeddings are how text enters the model. They are also how it leaves. After the last layer, the vector at the final position has to become a choice of next token. The model does this with one more table, used in reverse: it takes the dot product of the final vector with a vector for every token in the vocabulary, which gives one score per token. A higher score means the final vector points more in that token's direction. A softmax turns the scores into probabilities, and the model picks one.
This table is sometimes called the unembedding. In some models it is the input embedding table reused, on the reasoning that the direction that meant dog on the way in should mean dog on the way out, and sharing halves the parameters spent on vocabulary. Most large models today learn a separate output table instead. Either way, the model reads by looking up a vector and writes by measuring which vector it points toward. The list of numbers is the model's only language, at both ends.
Similarity is not truth. "Vaccines cause autism" and "vaccines do not cause autism" share a topic, a subject, and most of their tokens, so in most embedding models they land very close together. Negation is a small edit to the text and a small move in the space, even though it reverses the meaning. A semantic search for the first sentence will happily return the second. That is fine when the question is what a document is about, and useless when the question is whether it is right.
Embeddings inherit the biases of the text. In 2016 Tolga Bolukbasi and colleagues took the analogy arithmetic above and asked the widely used Google News word2vec vectors to complete "man is to computer programmer as woman is to". The answer was homemaker. The same vectors put nurse against surgeon and sewing against carpentry. The model had not been told any of that. It had absorbed a corpus in which those associations were frequent, and training turned frequency into geometry. Researchers who later tried to remove the offending direction found that it mostly hid the bias from the obvious test without removing it from the space.
Vectors from different models are not comparable. Each model learns its own space, with its own axes, its own scale, and its own number of dimensions. A vector from one model dropped into an index built with another is noise. Change your embedding model and every stored vector has to be recomputed, which is a real cost for anyone who has embedded a large collection.
And a vector is a summary, so it loses things. One list of numbers for a 500-word passage cannot hold every claim in it. Two passages that agree on their main topic and differ on a decisive detail can embed almost identically. Embeddings are good at "about the same thing" and weak at "says the same thing", and most of their failures in practice come from confusing the two.
Trust an embedding to tell you what a text is about. Do not trust it to tell you what the text claims, whether the claim is true, or whether two texts agree.
Every model Berges AI serves (Kimi, DeepSeek, GLM, and Qwen) is a transformer, so each message you send becomes a row of token vectors before anything else happens, and the reply comes back out through an unembedding table. Those tables are exactly as their makers learned them. The Berges layer of per-message instructions is not a fine-tune: it changes what the model reads, not the weights that turn tokens into vectors.
Try Berges AIIt depends on the model. Word2vec vectors usually had 100 to 300 numbers. The token embeddings inside large language models have a few thousand, and GPT-3 used 12,288. Dedicated embedding models for search typically return between 384 and about 3,000 numbers per text. More dimensions can hold finer distinctions but cost more to store and compare.
Yes. It is a list of numbers, and any embedding API will hand it to you as one. It means nothing by eye. The numbers are not labeled, and a single one on its own does not stand for anything. The only way to read an embedding is to compare it with other embeddings from the same model.
No. Every model learns its own space, with different axes, a different scale, and usually a different number of dimensions. Comparing a vector from one model with a vector from another produces meaningless numbers. If you switch embedding models, you have to re-embed everything you stored.
A database built to store embeddings and find the ones closest to a query vector quickly. It uses an index that avoids checking every stored vector, so it can search millions in milliseconds, usually with a small loss of exactness. It is the storage and search layer behind semantic search and retrieval-augmented generation.
No. A token is a chunk of text, like a word or part of a word, that the model has an ID for. An embedding is the list of numbers the model looks up or computes for that token. The token is the key and the embedding is the value, and the model does all its work on the value.
Every token you type is looked up in the model's embedding table as the first step, and each layer then rewrites those vectors using the surrounding context. At the end, the final vector is scored against a vector for every token in the vocabulary to choose the next token. Chat models do not return embeddings to you, but they run on them from the first step to the last.