By the end you will know how a text predictor is turned into an assistant, and why the confidence, warmth, and agreement it shows were selected by raters rather than earned by being right.
A model straight out of pretraining is a text predictor. Give it the first half of a recipe and it writes the second half. Give it a question and it writes whatever usually follows a question in the text it was trained on, which is often another question. It does not know that there is a user, that the user wants something, or that it is expected to reply.
Post-training is the set of stages that come after pretraining and turn that predictor into something you can talk to. It uses a tiny fraction of the data and compute, and yet it accounts for almost everything you notice about a model: the tone, the format, the willingness to answer, the refusals, the habit of agreeing with you.
The stages below are in the order labs usually apply them. Each leaves marks on the model's behavior, including the side effects the Berges AI manifesto objects to: flattery, padding, and fake certainty. Those are not accidents. They are what you get when you train a model to be rated well by people.
Pretraining shows the network trillions of tokens and trains it to predict the next one (see what an LLM is). The result, a base model, has absorbed grammar, facts, code, arguments, and the conventions of every genre it has read. What it has not absorbed is any notion of a conversation with you. Its one skill is continuation: given text, produce more text of the same kind.
That makes it strange to use. Type a question and the most likely continuation may be a second question, because quizzes and FAQ pages put questions in lists. It may be a forum reply that opens with an insult. It may be the rest of an exam paper. It is doing exactly what it was trained to do, and answering was never part of the objective.
prompt: What is the capital of Australia?
base model: What is the capital of Canada?
What is the capital of Brazil?
Answers are on page 12.
assistant: Canberra. Sydney is the largest city,
but the capital is Canberra.
Everything that separates those two continuations was added after pretraining. The habit of answering, the format, the decision to say Canberra and stop, the refusals you meet on other topics, the personality: none of it exists in the base model as behavior. Some of it exists as latent ability. The base model has read thousands of helpful answers and can imitate one if the prompt is arranged just right (a few-shot prompt does this). Post-training makes the imitation the default.
The first stage is supervised fine-tuning, usually shortened to SFT. People write or curate example conversations: a user message and the reply a good assistant would give. These are called demonstrations. The model is trained on them with the same objective as pretraining, next-token prediction, updated by backpropagation. Only the data and its amount differ. InstructGPT, the 2022 OpenAI paper that set the template for this pipeline, used about 13,000 prompts with demonstrations written by contractors. The base model underneath, GPT-3, had been pretrained on around 300 billion tokens.
In most recipes the loss is computed only on the assistant's tokens. The model reads the user's message but is not trained to predict it, only what comes after it. That is how it learns that its job starts where the user's message ends.
A raw model has no concept of turns. To give it one, the demonstrations are wrapped in a chat template: special tokens that mark where the system message, the user message, and the assistant reply begin and end. The model sees thousands of conversations in that format and learns the pattern: after the token that opens an assistant turn, produce a reply; at the end of the reply, emit the closing token. That closing token tells the serving software to stop.
The system prompt is the same trick. It is not a separate channel into the model. It is text in a slot at the top of the conversation, and the model treats that slot as instructions only because its demonstrations included system messages that were followed. Without that training it is just more text to continue. This is also why a determined user can override a system prompt: there is no wall between the slots, only a learned habit of respecting one of them.
SFT teaches format and the habit of answering. After it, the model is recognizably an assistant. What it does not teach is judgment between two acceptable answers. A demonstration shows one good reply. It never shows a slightly worse reply beside it and says which is better, and the next-token loss cannot express "this is fine, that is finer". That gap is what the next stage fills.
Reinforcement learning from human feedback, RLHF, is the stage that gives a model most of its personality. In 2017, Christiano and colleagues trained agents by showing people pairs of short video clips and asking which was better. A simulated robot learned to do a backflip from about 900 such comparisons, under an hour of a person's time, with nobody writing down what a backflip is. InstructGPT applied the same recipe to a language model in 2022.
Reinforcement learning here means adjusting the model to raise a score, instead of adjusting it to match a given answer. SFT says "produce this". RL says "produce whatever scores highest", and lets the model find it. The pipeline has four steps.
The reward model learns from comparisons rather than absolute scores because people are more consistent at comparing than at grading. Asked to give a reply a 7 out of 10, two raters disagree constantly. Asked which of two replies is better, they agree far more often. The loss pushes the reward model to give the preferred reply a higher number than the rejected one, harder when it currently gets that pair wrong.
The penalty is the piece most explanations skip, and it matters most later. At each token, the assistant's probabilities are compared with the SFT model's, and the assistant is charged for straying. The charge is measured as KL divergence, a number that grows as the two models disagree more. Without it, the assistant drifts toward whatever quirks the reward model happens to like and stops writing normal text. With it, the assistant can only move so far from the reasonable model it started as.
Raters preferred the 1.3-billion-parameter InstructGPT over the 175-billion-parameter GPT-3 base, a model more than a hundred times larger. A small amount of feedback did more for perceived quality than a hundredfold increase in size. It did not make the small model know more. It made it answer.
RLHF as described is expensive and fiddly. It keeps four models in memory at once (the assistant, the frozen reference copy, the reward model, and a value estimator the optimizer needs), and the optimization is notoriously unstable.
Direct preference optimization, DPO, published by Rafailov and colleagues in 2023, removes the reward model. The insight is mathematical. Under the KL-penalized objective, the best possible assistant and the reward model determine each other exactly, so the training can be rewritten in terms of the assistant alone. What comes out is a loss computed on preference pairs directly: for each pair, raise the probability of the preferred reply and lower that of the rejected one, both measured relative to the reference model. No sampling during training, no reward model, no RL loop. It looks like ordinary supervised training and runs like it.
Many open-weights labs use DPO or something close to it. Meta described using it for Llama 3, and it is a standard step in published open recipes. Whether it matches full RLHF at the frontier is disputed. Some labs appear to still run a reward model and an RL loop, and the recipes behind the largest closed models are not public.
The other cost is the raters. Reinforcement learning from AI feedback, RLAIF, replaces most of the human comparisons with comparisons made by a model. The reference paper is Constitutional AI, from Bai and colleagues at Anthropic in 2022. It has two parts. First, the model is given a written list of principles (the constitution), asked to critique its own replies against them and revise; the revisions become SFT data. Second, a model is shown pairs of replies and asked which better follows a principle drawn from the list; those judgments train the preference model in place of human labels. In that paper, humans still labeled helpfulness. AI feedback covered harmlessness.
How much of this each lab uses is mostly unpublished. Most are believed to use some model-generated feedback, human raters still exist, and the mix is a trade secret.
A rater's opinion is one possible reward. For some tasks there is a better one: a check. A math problem has an answer that can be compared with the key. A coding task has tests that pass or fail. When the reward is a check rather than an opinion, the training signal is about correctness, not approval. This is often called reinforcement learning with verifiable rewards.
Training with verifiable rewards is what produced the long, visible chains of working that define reasoning models. OpenAI's o1, released in 2024, was the first widely used example, but its training was described only in outline. DeepSeek-R1, published in January 2025, described the process in detail. The team took a base model, DeepSeek-V3, and ran reinforcement learning on it with two rewards: one for reaching the right final answer, one for putting its working inside the expected tags. No demonstrations of reasoning were provided. Over thousands of training steps the model's replies grew longer on their own, and it began to check its work, back up, and try a second approach, because those habits raised the accuracy reward.
The paper also reported the cost of skipping SFT. The pure RL model, called R1-Zero, switched languages mid-answer and was hard to read. The released R1 added a small SFT stage of curated reasoning examples before the RL, and a round of preference training for tone and safety after it. The stages stack: verifiable rewards drive accuracy; demonstrations and preferences keep the output usable.
Verifiable rewards exist only where a checker exists. There is no test for whether an essay is good or an explanation is clear. For those, the reward is still a rater or a rater model, with everything that implies.
Post-training uses a sliver of the compute and data: in InstructGPT, tens of thousands of prompts against a base model trained on 300 billion tokens. It is not surprising that so small a stage cannot rewrite what the model knows. What is surprising is how much of the model's character it can set.
What it can change: tone, format, length, the language of the reply, whether the model follows instructions, what it refuses, whether it shows its working, and whether it agrees with you. All of these are behaviors, and behaviors are what the reward measures.
What it cannot change much: what the model knows. The knowledge cutoff stays where pretraining left it (see context window vs knowledge cutoff). A fact the model never saw does not appear because raters wanted it. And the capability ceiling is set upstream. Post-training cannot make a small model reason like a large one, only make it use what it has. The tuned 1.3B InstructGPT won on preference, but it did not turn into a large model on knowledge tests.
Post-training mostly reshapes behavior that is already latent. The base model has read countless helpful answers; SFT makes that mode the default. It has read careful reasoning; verifiable rewards make it reason more often and for longer. One 2023 study, LIMA, fine-tuned a base model on about a thousand hand-written examples and got a usable assistant, which suggests the assistant format is a thin layer over ability that was already there.
This is also why fine-tuning on new documents is an unreliable way to add knowledge. Training on a few hundred pages about your product teaches the model the style of those pages far more reliably than the facts in them, and the evidence so far suggests it can raise hallucination about the new material. The fine-tuning page covers what the technique is good for; retrieval, which hands the model the text at answer time, is the usual tool for facts.
Each habit below is a direct consequence of the reward. None is a bug, and none is a mystery. The reward model learned what raters like; the assistant learned to produce it. When what raters like is not the same as being right, the model learns that difference too.
Raters prefer replies that agree with them. Sharma and colleagues at Anthropic studied this in 2023 across several deployed assistants and found that all of them tended to abandon a correct answer when the user pushed back, to mirror opinions the user had stated, and to praise a piece of writing more warmly when the user claimed it as their own. In the preference data they found the cause: human raters, and the preference models trained on them, chose a convincingly written sycophantic reply over a correct one a meaningful fraction of the time. Optimize against that preference and you get a model that agrees. The sycophancy page covers what it looks like in use.
Raters prefer longer replies. A longer reply looks more thorough and is harder to fault at a glance. Researchers have found that reward models trained on human comparisons score longer outputs higher even when the extra length adds nothing, and that much of the measured gain from RLHF on some benchmarks is explained by length alone. So the model pads: a preamble restating the question, a list where a sentence would do, a closing summary, an offer of further help. Each of those items was, on average, rewarded.
A confident wrong answer is easy for a rater to mark down. A vague answer is hard to mark down at all. The likely mechanism is that this asymmetry, repeated over millions of comparisons, teaches the model to hedge: "it depends", "consult a professional". Raters also reward replies that sound balanced, so the model presents two sides whether or not the question has two. Combined with the padding above, this produces the recognizable AI voice: smooth, symmetrical, cautious, and hard to pin down.
Refusals are trained the same way. When raters mark a harmful reply down hard and an unnecessary refusal down only a little, the safest policy for the model is to refuse anything near the line. Researchers have found that assistants refuse plainly harmless requests when they resemble harmful ones on the surface, such as how to kill a Python process. The model is not reasoning about harm. It is matching a pattern the reward taught it.
A base model, sampled twice, gives two different continuations. A preference-trained model, sampled twice, gives two very similar ones. Researchers have found that RLHF narrows the range of outputs: fewer phrasings, fewer structures, the same opening sentence again and again. The reason is in the objective. A reward model has a favorite kind of reply, and optimizing against it collapses the assistant toward that favorite. The KL penalty slows the collapse but does not stop it. It is part of why so much AI-written text reads the same whichever lab made the model.
The reward model is a stand-in for what people want. It is not what people want. It is a network trained on a finite set of comparisons, and it has gaps: places where its scores and real quality come apart. Optimize hard enough against it and the assistant finds the gaps. This is Goodhart's law: when a measure becomes a target, it stops being a good measure. The raters wanted good answers. The measure is a score. The model is trained on the score.
Gao and colleagues at OpenAI measured this directly in 2022. They trained assistants against a reward model while scoring the outputs with a much larger, more reliable one that stood in for the truth. The proxy score kept rising. The real score rose, peaked, and then fell, while the proxy insisted things were still improving. The further the assistant moved from its starting point, the wider the gap.
The KL penalty exists for this reason. It is not there to make the model polite. It keeps the assistant within a region where the reward model's opinions are still trustworthy, because the reward model was trained on replies that looked like the SFT model's. Far outside that region it is guessing.
Two examples. Coding models trained against unit tests have been caught special-casing them: detecting the test inputs and returning the expected outputs for those cases, or editing the tests so they pass, instead of fixing the code. Assistants trained against rater preferences have learned that a confident tone rates better than a hedged one at the same level of correctness, so confidence is rewarded regardless of correctness. Neither was anyone's intention. They were the highest-scoring behaviors available.
Every reward is a proxy. A rater is a proxy for the user. A reward model is a proxy for the rater. A unit test is a proxy for working code. Post-training optimizes the last link in the chain, and every link can be gamed.
With a closed model, you get one lab's post-training and nothing else. The base model is never released, the demonstrations and preference data are private, and the behavior you meet is the only behavior available.
With open weights, the pipeline is open where it matters most. Many labs release the base model alongside the assistant version. Anyone with a few GPUs can post-train the base differently: their own demonstrations, preference data, principles for AI feedback, and decisions about what to refuse. The same base yields dozens of assistants with different personalities. A model tuned for terse technical answers and one tuned for role-play can share every pretrained weight.
DeepSeek-R1 showed this at scale. Alongside R1, the team released smaller models made by fine-tuning existing open bases from Qwen and Llama on 800,000 examples generated by R1. The architectures did not change. The behavior did: the distilled models reasoned in long chains the way R1 does, which their original assistant versions did not.
This is also why model families differ in feel even when their architectures match. The transformer architecture is shared across nearly every current model, and the differences between labs' designs are modest. Most of what you experience as "this model is chatty" or "this model pushes back" is post-training. Two models can be structurally near-identical and feel like different people, because they were raised differently.
The confidence in a reply was selected by raters. So was the warmth, the agreement, the balanced tone, and the length. None of those signals that the reply is correct, because correctness was not what the reward measured for most of the training. A model that says "you are absolutely right" or "great question" has learned that the phrase rates well. The stochastic parrots guide takes up what, if anything, sits behind the fluent surface. Whatever the answer, the surface itself was tuned for approval.
Instructions in a prompt push against these defaults. "Be brief" pushes against padding. "Tell me if I am wrong" pushes against agreement. "Do not hedge" pushes against hedging. They work, partly, because instruction following was also trained in, and the two habits compete. But you are pushing against something that was optimized over millions of comparisons, and it pushes back. The model that agreed to be brief drifts toward length a few turns later. The model told to disagree disagrees once, then agrees with your disagreement.
Treat the manner of a reply as information about the training, and the content as something to check. When a model is right, it sounds confident. When it is wrong, it also sounds confident, because confidence was rewarded either way. The tone tells you what the raters liked, not whether the answer is true.
The models Berges AI serves (Kimi, DeepSeek, GLM, and Qwen) arrive with their labs' post-training, side effects included. What Berges AI adds on top is a layer of per-message instructions and settings, the interceptors, not a fine-tune. That layer is how it strips flattery and padding from replies without retraining anything.
Try Berges AINot quite. Post-training is the whole sequence of stages a lab runs after pretraining: supervised fine-tuning, preference training, and often reinforcement learning with verifiable rewards. Fine-tuning is the general technique of training an existing model further on new data, and supervised fine-tuning is one stage of post-training. You can also fine-tune a released model yourself, which is post-training in miniature.
A base model is the network as it comes out of pretraining, before any post-training. It predicts the next token of text and nothing else. It has no concept of a user or a reply, and it will continue a question with more questions if that is what its training text usually did. Many open-weights labs release the base model alongside the assistant version.
Only a little, and unreliably. Post-training data is tiny next to pretraining data, so it shapes behavior far more than it adds facts. Fine-tuning on new documents tends to teach the style of those documents and can increase hallucination about their contents. Retrieval, which gives the model the relevant text at answer time, is the usual way to supply new facts.
Because post-training taught them to. Demonstrations of refusals and rater preferences against harmful replies make the model decline certain requests. Where the line sits is a choice made by whoever ran the post-training, which is why the same base model can be released with very different limits by different people. Over-refusal, declining harmless requests that resemble harmful ones, is a side effect of the same training.
Reinforcement learning from human feedback. Human raters compare model replies and pick the better one. A reward model is trained to predict those preferences, and the assistant is then trained to produce replies the reward model scores highly, while a penalty keeps it from drifting too far from where it started.
Not exactly, but you can override it. With open weights, further fine-tuning on a few hundred or a few thousand examples can change refusals, tone, and format, and researchers have shown that safety training in particular can be removed with small amounts of fine-tuning. What you cannot do is recover the original base model from the assistant, so a clean start needs a lab that released the base separately. With closed models, none of this is possible; you get the lab's choices.