How a transformer goes from random weights to a language model — loss functions, gradient descent, scaling laws, and what training dynamics reveal about learning itself.
Before a single gradient is computed, there is a dataset: trillions of tokens scraped, filtered, and assembled from across the web and curated corpora. The quality, mixture, and scale of this data are arguably as important as the model architecture — they determine what the model can learn to predict, and therefore what it will know.
Modern LLMs train on a heterogeneous mixture of text domains, each contributing different linguistic and factual content. Common Crawl — a nonprofit snapshot of the public web — is the dominant source, but it requires heavy filtering before use. Curated sources like books, Wikipedia, and code provide higher signal-to-noise ratio but are orders of magnitude smaller.
Raw Common Crawl is noisy — full of spam, boilerplate, duplicate content, and non-English text. A preprocessing pipeline runs before any tokenization:
Remove pages below a minimum word count, with a very high fraction of punctuation or digits, or with repeated n-grams (a hallmark of boilerplate). C4 relied on rules like these alone — keep only lines ending in punctuation, drop pages with "lorem ipsum" or code braces (Raffel et al. 2020). CCNet added perplexity scoring: a small 5-gram language model trained on Wikipedia scores each page, and pages with low perplexity — Wikipedia-like text — form the high-quality "head" that later datasets keep (Wenzek et al. 2020).
Exact and near-duplicate paragraphs are removed using MinHash locality-sensitive hashing. Deduplication matters because memorized content does not generalize and can inflate benchmark scores — training on the same document twice is not twice as useful.
If evaluation benchmarks appear in the training corpus, measured performance overstates true capability. Labs now perform n-gram overlap checks between training data and benchmarks and publish contamination analyses alongside results.
After filtering, documents are concatenated and cut into fixed-length chunks of T tokens (typically 2048–4096). Each chunk becomes one training sequence. Because the objective is to predict every next token, a chunk of T tokens yields T − 1 training examples — input x1:t predicts target xt+1 for every position t.
The fraction of training tokens from each domain is a critical hyperparameter. Upsampling books and Wikipedia relative to raw web text improves factual accuracy and writing quality; adding code gives modest gains even on non-code reasoning tasks (see the box below). Upsampled sources are seen more than once: GPT-3 went through Wikipedia ~3.4 times but through filtered Common Crawl less than half a time. Most labs tune these proportions empirically on downstream benchmarks.
| Domain | Typical range | Key contribution |
|---|---|---|
| Web (Common Crawl) | 50–80 % | Broad world knowledge, conversational style, diverse topics |
| Books | 5–15 % | Long-range coherence, narrative structure, formal prose |
| Wikipedia | 2–5 % | Factual density, encyclopaedic coverage, citation norms |
| Code (GitHub) | 4–12 % | Logical structure, step-by-step reasoning, formal grammars |
| Papers (arXiv etc.) | 2–5 % | Technical depth, mathematical notation, scientific reasoning |
| Curated other | 1–5 % | Dialogue data, legal text, multilingual documents |
In controlled pretraining ablations (470M–2.8B parameters), adding code improved natural-language reasoning by up to 8.2% and world knowledge by 4.2% (relative), compared with text-only training (Aryabumi et al. 2024). The gains are real but modest, and why they happen is not settled; one hypothesis is that code's explicit, step-by-step structure transfers to other tasks.
Training token counts have grown by more than 10× every two years. GPT-2 was trained on WebText, ~40 GB of text from ~8M web pages (OpenAI did not publish a token count; roughly 9–10B tokens per pass is a common estimate); GPT-3 used ~300B tokens; LLaMA 1.4T; LLaMA 3 15T. The figure below shows total pretraining tokens for selected models:
For a fixed compute budget and architecture, data curation decisions — which sources to include, at what mixture, with which filters — can change final model quality substantially. This is why data pipelines are now treated as proprietary intellectual property.
Training a language model reduces to a single objective: make the model's predicted probability for the next token as high as possible — across every position in every training document. Everything else — backpropagation, adaptive learning rates, gradient clipping — is machinery in service of this one goal.
Every training step executes the same seven-stage cycle, repeated millions of times across the corpus:
A batch is B chunks processed simultaneously in a single forward-backward pass. Running chunks in parallel saturates GPU cores; averaging the loss over B chunks gives a more stable gradient estimate than any single chunk alone. Typical LLM batch sizes are large — 1,024 chunks × 4,096 tokens ≈ 4M tokens per step.
One step = one batch → one weight update. One epoch = enough steps to have seen every chunk in the dataset once. LLM pretraining typically covers about one pass over the data, and the "epoch" differs by source: GPT-3 saw Wikipedia ~3.4 times but filtered Common Crawl only ~0.44 times (Brown et al. 2020). Practitioners track progress in steps or tokens seen, not epochs.
Surprise — also called self-information — is a formal quantity in information theory, defined as −log p(x) for a single outcome x. The definition captures the intuition directly: a high-probability event carries low surprise; a low-probability event carries high surprise. It is precisely the quantity that appears in the loss for each individual token prediction.
Self-information is the building block of a family of related concepts that appear throughout LLM training:
The expected surprise under a distribution — E[−log p(x)]. A measure of how uncertain a distribution is on average. A peaked distribution has low entropy; a uniform one has high entropy.
The expected surprise when model distribution q is used to encode events drawn from true distribution p — Ep[−log q(x)]. Averaging the per-token loss over the dataset gives exactly this quantity.
For each position in a sequence, the model outputs a probability distribution over the vocabulary. The per-token loss measures how surprised the model is by the actual next token:
If the model assigns probability 0.9 to the correct token, the loss is −log(0.9) ≈ 0.105. If it assigns only 0.01, the loss is −log(0.01) ≈ 4.6. The model is penalized in proportion to its surprise.
Log-probabilities are additive across positions, making the loss the negative average log-probability over the sequence. Cross-entropy equals the entropy of the true distribution plus the KL divergence from it to the model — since that entropy is constant with respect to model parameters, minimizing cross-entropy is equivalent to minimizing KL divergence and to maximum likelihood estimation.
The quantity the optimizer actually minimizes is the per-token loss averaged over all positions and all sequences in the batch:
where B is batch size and T is sequence length. (Notation: the loss is written "Loss" throughout this lesson; L is reserved for the number of layers.) Gradients of this scalar with respect to every weight θ are computed by backpropagation, then used to update the weights.
No human labels are needed. The next token in any text document is the supervisory signal. This is why scaling to trillions of tokens is feasible — the internet provides essentially unlimited training data for free.
Perplexity exponentiates the average loss Loss, giving an intuitive measure of how many tokens the model is "confused between" at each step on average:
Because Σ log P = log(Π P), the average loss is −(1/(B·T)) · log(Π P), and exponentiating gives PPL = (Π P)−1/(B·T) — the geometric mean of 1/Pt across all token positions, or equivalently 1 divided by the geometric mean of Pt. A perplexity of 20 means the model's geometric-mean confidence in each correct token was 1/20 ≈ 5%. The geometric mean is the right notion here because sequence probability is a product of conditionals — averaging in log-space is geometric-mean in probability space.
On WikiText-2, without any fine-tuning, GPT-2 reached a perplexity of 29.4 (117M parameters) down to 18.3 (1.5B) (Radford et al. 2019, Table 3). GPT-3's often-quoted ~20 (20.5) is on a different benchmark, Penn Treebank (Brown et al. 2020). Perplexity is also only comparable between models that share a tokenizer: it is measured per token, so splitting the same text into more, smaller tokens changes the number. Modern models reach single digits on some benchmarks. A model predicting uniformly over GPT-2's 50,257-token vocabulary has loss ln(50,257) ≈ 10.82 and PPL = 50,257.
We hold out a validation set never seen during training. Every N steps, a forward pass is run on this held-out data with gradients disabled — no weight update follows. The result is purely diagnostic: the gap between training and validation loss reveals how well the model generalizes. A widening gap means the model is memorizing training data rather than learning to generalize. Practitioners use it to decide when to stop training, decay the learning rate, or select which checkpoint to keep. Modern LLMs trained on trillions of tokens typically show minimal gaps, because the corpus is so large. A small gap does not rule out memorization, though: large models still reproduce some training text verbatim, more so for bigger models and for text that is duplicated many times in the corpus (Carlini et al. 2023).
Backpropagation is the chain rule of calculus applied systematically to compute ∂Loss/∂θ for every weight θ in the network. The key insight: because the forward pass is a composition of differentiable operations, gradients flow backwards through the same computational graph.
Before getting into backpropagation, it helps to recall a basic result from calculus. If Loss depends on an intermediate value y, which itself depends on x, then:
The gradient of Loss with respect to x equals the gradient of Loss with respect to y, multiplied by the gradient of y with respect to x. That's it — this single rule is the entire mathematical foundation of backpropagation. Everything else is applying it repeatedly through a computation graph.
A transformer is a stack of residual blocks. At each layer l, the residual stream is updated by:
Here xl is a vector of dimension dmodel — the accumulated state of computation up to layer l. Fl is whatever the block computes (attention or MLP), and its output is the delta added to the stream. The next layer's input xl+1 is simply the current stream plus that delta.
To propagate gradients backward, apply the chain rule with y = xl+1 and x = xl:
The second term requires differentiating xl+1 = xl + Fl(xl) with respect to xl. Because xl appears in both terms — directly in the skip connection, and as input to Fl — the derivative is:
Substituting back gives the key backpropagation equation for a residual network:
This equation is applied as a backward recurrence: starting from ∂Loss/∂xL at the output layer (where the loss is computed), each step produces the gradient one layer earlier — ∂Loss/∂xL−1, then ∂Loss/∂xL−2, all the way back to the input. That propagation from deep to shallow is the "back" in backpropagation.
The identity term I means that even if ∂Fl/∂xl is nearly zero — small weights, saturated activations — the gradient still passes through essentially unchanged: ∂Loss/∂xl ≈ ∂Loss/∂xl+1. Without the skip connection, you would multiply L Jacobians in a chain, and their product typically vanishes.
The recurrence needs a starting value: ∂Loss/∂xL at the final layer. This is computed directly from the loss, not from another backward step. After the last residual stream, the model applies an unembedding step and softmax to produce a probability distribution over the vocabulary:
where y is the index of the correct next token. (For simplicity we leave out the final LayerNorm that GPT-2 applies before the unembedding; including it adds one more chain-rule step.) To find the gradient ∂Loss/∂zk for each logit, first rewrite the loss by expanding log py using the softmax definition py = exp(zy) / Σj exp(zj). Taking the log and applying log(a/b) = log a − log b:
Negating both sides gives the loss:
Differentiating with respect to logit zk:
In the first term, ∂zy/∂zk is 1 if k = y and 0 otherwise — the indicator 𝟙k=y. The second term is pk by definition of softmax. So:
In vector form: ∂Loss/∂z = p − 𝟙y. For the correct token the gradient is py − 1 (negative — push the logit up); for all wrong tokens it is pk (positive — push them down). When the model is already confident and correct, py ≈ 1 and the gradient nearly vanishes — little correction needed. One more chain rule step through the unembedding gives the seed:
This dmodel-dimensional vector is handed to the recurrence as ∂Loss/∂xL, and the backward pass proceeds from there.
The formula is self-regulating: plugging in numbers makes this concrete.
| Scenario | Gradient | Effect |
|---|---|---|
| Wrong token, pk = 0.40 | +0.40 | Push logit down — proportional to how much probability it was given |
| Correct token, uncertain (py = 0.05) | −0.95 | Push logit up strongly — model was nearly ignoring the right answer |
| Correct token, confident (py = 0.97) | −0.03 | Nearly zero — model already knows this, no significant update needed |
Fig. 7 showed one failure mode — vanishing gradients — and one solution. But gradient pathologies come in several flavors, and transformers have a specific fix for each. The table below maps each problem to its symptom and the architectural choice that addresses it.
| Problem | Symptom | Transformer Solution |
|---|---|---|
| Vanishing gradients | Early layers learn nothing; loss stalls | Residual connections + LayerNorm |
| Exploding gradients | Loss spikes to NaN | Gradient clipping (clip_grad_norm ≤ 1.0) |
| Attention saturation | Softmax outputs near 0 or 1; gradients near 0 | Scaling QK by 1/√dk |
| Rank collapse | Stacked attention makes every token's vector nearly identical, so positions can't be told apart | Residual connections and MLPs (Dong et al. 2021) |
Classic initialization schemes such as Xavier/Glorot and He weren't designed to make the forward pass look tidy — they were derived by asking what weight scale keeps gradient variance from exploding or vanishing as it propagates backward through L layers. Initialization is a gradient-flow argument first. The first backward pass is computed from the initial weights, so a bad start can trigger the exact pathologies in the table above before training has had any chance to correct them.
Why not zero? Every neuron in a layer computes the same output and receives the same gradient — the network never breaks symmetry and every layer collapses to a single effective direction.
Why not large random values? Activations (and their gradients) grow or shrink exponentially with depth, producing NaN losses from the very first pass.
The solution is to scale random draws so that gradient variance stays roughly constant from layer to layer. Two classic schemes come from earlier deep networks:
Draw weights from a distribution with σ = √(2 / (fanin + fanout)). Designed for layers with saturating activations (tanh, sigmoid).
Draw weights with σ = √(2 / fanin). Designed for ReLU, which zeroes about half of its inputs.
GPT-2 and most LLMs use neither. They draw every weight matrix from a normal distribution with a fixed σ = 0.02. For comparison, with dmodel = 768 Xavier would give σ = √(1/768) ≈ 0.036, so 0.02 is a more conservative choice that keeps initial activations smaller.
GPT-2 adds one more tweak: the output projection of each residual block is scaled down by an extra factor of 1/√(2L), where L is the number of layers. Since every block adds to the residual stream, this keeps the total variance of the stream from growing as depth increases.
After a few thousand gradient steps the optimizer has moved weights far from their starting values, and the exact initialization scheme is largely forgotten. Its lasting effect is on the earliest steps — which is why large training runs also use a learning rate warmup: the learning rate ramps from near-zero over the first few hundred to few thousand steps (Tab 4), giving the model time to settle into a stable regime before taking large gradient steps.
GPT-2 small has ~124M parameters: ~39M in the token and position embeddings, and ~85M in its 12 blocks, split across:
WQ, WK, WV, WO per head per layer. These learn what to attend to — which token-pair relationships are useful for prediction.
Wup, Wdown (and Wgate in gated models). These account for ~2/3 of the block parameters and learn factual associations — often thought of as a key-value memory.
The forward pass stores activations for later gradient computation — for a 100-layer model, this requires storing 100 activation tensors simultaneously. Gradient checkpointing discards intermediate activations and recomputes them during the backward pass, trading compute for memory. This is an engineering trick for making training feasible on memory-limited hardware; it is not essential to understanding how LLMs work.
LLMs and backpropagation are a proof of concept that predictive learning from raw data can produce rich representations — but the learning rule is almost certainly different from what the brain uses. The representations may converge even when the learning rules diverge, which is both fascinating and a challenge for using LLMs as neural models.
Gradient descent tells us the direction to update weights. Optimizers determine the step size and adaptation — how much to trust each gradient and whether to accumulate momentum.
SGD, Momentum, and Adam are three separate algorithms — not steps in a single process. Each one was invented to fix a specific limitation of the previous. In practice you pick one; modern LLM training uses Adam (or its variant AdamW).
θ (theta) represents all the weights in the model — the millions or billions of numbers that get updated during training. It is treated as a single symbol here for compactness, but the update rule applies independently to every individual parameter.
← (left arrow) means assignment, not equality. θ ← θ − η·g reads as: "replace θ with the result of θ minus η times g." Both sides refer to θ, but at different moments — the right side uses the current weights, the left side stores the updated weights. Writing = instead would be a false mathematical statement; the arrow makes clear this is an operation that happens over time.
The simplest possible optimizer: move the weights a small step in the direction opposite to the gradient.
η is the learning rate — a scalar that controls step size. g is the gradient of the loss with respect to θ, computed by backpropagation.
Limitation: Each step ignores all previous gradients. On loss landscapes with long, narrow valleys — common in deep networks — SGD oscillates back and forth across the valley walls instead of moving efficiently toward the minimum.
Fixes SGD's oscillation problem by maintaining a velocity term v that accumulates past gradients — like a ball rolling downhill that builds speed in consistent directions and dampens oscillations across them.
β (typically 0.9) controls how much past gradients contribute. A gradient in the same direction as v amplifies it; a gradient pointing the opposite way partially cancels it out. The net effect: smooth acceleration along the valley floor, reduced bouncing across it.
Limitation: The step size η is the same for every parameter in the network. But different weight matrices have very different gradient magnitudes — some are large and consistent, some are tiny and noisy. A single global learning rate is always too large for some parameters and too small for others.
Fixes momentum's one-size-fits-all learning rate by giving each parameter its own effective step size. It tracks two running averages for every weight: the mean gradient (first moment m) and the mean squared gradient (second moment s).
Typical defaults: β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸.
The hats (m̂, ŝ) denote bias correction — without it, the running averages start near zero and the first few steps would be too large. Concretely: m̂ = m / (1 − β₁ᵗ), where t is the current step. At step 1 with β₁ = 0.9, the denominator is only 0.1, so m̂ is 10× larger than the raw m — exactly compensating for the cold start. The correction fades as t grows and β₁ᵗ → 0.
The effective step size for a parameter is η · m̂ / (√ŝ + ε), so the step is large when m̂ is large and/or ŝ is small. What actually drives step size is the signal-to-noise ratio m̂/√ŝ: a parameter with large, consistent gradients has both large m̂ and large ŝ, and the two partially cancel; a parameter with small but consistent gradients has small m̂ and small ŝ, so it still gets a meaningful step; a parameter with noisy gradients has small m̂ (noise cancels in the mean) but large ŝ, so it gets a small step.
With gradients arriving from 124M (GPT-2 small) to 1.5B (XL) parameters, different weight matrices have wildly different gradient magnitudes. Adam normalizes them by √ŝ, making the optimizer robust to this heterogeneity. The gradient mean (m) also smooths over mini-batch noise, reducing sensitivity to learning rate choice. The second moment (s) provides per-parameter normalization: Adam divides each parameter's update by the running RMS of its own gradient, so the step size is set roughly by the learning rate and by how consistent the gradient's sign is, not by its raw magnitude. This evens out step sizes across parameters whose gradient scales differ by orders of magnitude.
The variant used in practice decouples weight decay from the gradient update:
where λ is the weight decay coefficient (typically 0.1). This penalizes large weights directly, acting as an implicit regularizer without interfering with Adam's gradient normalization. Before each update, gradient clipping rescales all gradients so their global norm never exceeds a threshold c — preventing a bad batch from causing a runaway parameter update.
The learning rate is scheduled throughout training in two phases, the standard schedule used for GPT-3 and Chinchilla:
Warmup prevents early training instability when gradient estimates are noisy; it typically lasts a few hundred to a few thousand steps, a small fraction of the run. Cosine decay then gradually reduces the learning rate along half a cosine wave, allowing the optimizer to settle into a good minimum rather than oscillating around it. Some recent runs use a variant instead: hold η constant for most of training, then drop it quickly to near zero in a short final "cooldown" (warmup-stable-decay). This makes it easy to continue training later from the constant phase.
| Symbol | Typical value | Effect |
|---|---|---|
| η (learning rate) | 10−4 to 3 × 10−4 | Primary knob — too high → instability; too low → slow convergence |
| T (context length) | 512–8192 tokens | Maximum span a token can attend to; multiplies with batch size to set total tokens per gradient step |
| B (batch size) | 256–4096 sequences | Number of sequences per gradient update — larger batches reduce gradient noise; often scaled with learning rate |
| β1 | 0.9 | First moment decay — momentum smoothing |
| β2 | 0.95 or 0.999 | Second moment decay — controls gradient normalization memory |
| λ (weight decay) | 0.1 | Implicit regularization — prevents weight explosion |
| c (grad clip norm) | 1.0 | Caps gradient norm — prevents exploding gradient catastrophe |
One of the most striking findings in deep learning: model performance improves predictably as a power law of compute, data, and parameters. These empirical relationships, discovered around 2020, fundamentally changed how AI labs plan training runs.
OpenAI's initial scaling law paper established that language model loss follows a power law in parameter count, data, and compute — holding across 7 orders of magnitude. For parameters alone:
where N is the (non-embedding) parameter count (≈ 12·L·dmodel2 for a transformer with L layers and hidden size dmodel), Nc ≈ 8.8 × 1013 is a fitted scale constant, and αN ≈ 0.076. Kaplan's pure power law has no irreducible-loss term; a separate law Loss(D) = (Dc/D)αD with αD ≈ 0.095 describes data size. Notice that D — the number of training tokens — does not appear in the parameter law at all; training data size was treated as effectively unlimited. From this, Kaplan et al. concluded that for a fixed compute budget, the priority should be maximizing model size N — even at the expense of training on fewer tokens. This drove the GPT-3 era design philosophy. It turned out to be wrong.
DeepMind's Chinchilla paper superseded the Kaplan prescription. By training hundreds of models at systematically varied (N, D) combinations — rather than holding one fixed — they obtained a cleaner joint fit:
where D is the number of training tokens, E ≈ 1.69 is the irreducible loss (the entropy of natural text, in nats per token), A and B are fitted constants, α ≈ 0.34, and β ≈ 0.28. Because α ≈ β, neither term dominates — starving either N or D hurts equally. Each training token costs roughly 6N FLOPs (2N forward + 4N backward), so a fixed compute budget C ≈ 6ND creates a direct tradeoff: doubling model size forces halving training tokens. Minimizing Loss(N, D) subject to this constraint yields N* ∝ Cβ/(α+β) and D* ∝ Cα/(α+β). With α ≈ 0.34 and β ≈ 0.28 this gives N* ∝ C0.45 and D* ∝ C0.55; Hoffmann et al.'s headline result, combining this fit with two other methods, rounds both to ≈ C0.5 — model size and data should grow at about the same rate. The specific ratio D*/N* requires the empirically fitted constants A and B; Hoffmann et al. evaluated it across hundreds of training runs and found roughly 20.
Llama and successors deliberately overtrain smaller models beyond Chinchilla-optimal — because inference cost matters too. A smaller model at the same loss is cheaper to serve at scale, so training longer on more data makes economic sense even if "suboptimal" for training compute.
Some abilities — like multi-step arithmetic, chain-of-thought reasoning, multi-hop reasoning, and instruction-following — have been reported to appear suddenly at threshold model sizes rather than improving smoothly (Wei et al. 2022). They seem absent below a threshold, then functional above it. These emergent phenomena are not captured by loss curves, the boundary is fuzzy and depends on evaluation metric, and they remain poorly understood theoretically.
The apparently sudden appearance of emergent phenomena may be an artifact of coarse evaluation metrics (Schaeffer et al. 2023). When measured with finer-grained partial-credit scoring, the same capabilities often improve smoothly with scale. Distinguishing genuine discontinuities from measurement failures remains an open methodological problem.
The loss curve is not just a number that decreases. Its shape, speed, and irregularities reveal what the model is actually learning and when — sometimes in surprising ways.
A healthy training run shows:
Discovered in 2022 by Power et al.: models trained on algorithmic tasks (modular arithmetic, permutations) sometimes first overfit the training data, then — after a long plateau — suddenly generalise to the validation set. This "delayed generalisation" can occur long after training loss reaches near-zero — in some runs, around a thousand times as many steps later.
The classic form of grokking needs a training set small enough for the model to memorize it fully (training loss → 0). That is easy to arrange in controlled algorithmic tasks but is not the situation in large-scale LLM pretraining, where most text is seen about once. So the overall pretraining loss curve of an LLM looks like the normal scenario, not like grokking. Whether grokking-like delayed generalisation happens for particular skills or subsets of the data inside LLMs is an open research question.
The sudden jump in validation accuracy hides a gradual process. Nanda et al. (2023) reverse-engineered a small transformer doing modular arithmetic and split training into three continuous phases: memorization (a lookup-table-like solution), circuit formation (a compact Fourier-based algorithm that actually computes the answer grows slowly underneath, while training loss is already near zero) and cleanup (weight decay removes the memorizing components). Validation loss drops sharply only at cleanup, once the memorized solution stops masking the general one. The same task ends up solved by a qualitatively different internal mechanism than the one the model started with.
During long training runs, the loss sometimes jumps suddenly — a "loss spike" — before recovering. Common causes:
| Cause | Signature | Fix |
|---|---|---|
| Bad data batch | Single large spike, quick recovery | Better data filtering; gradient clipping |
| η too high | Periodic spikes, slow recovery | Reduce peak η; increase warmup |
| Batch size change | Spike at schedule transition | Smooth transitions; adjust η linearly |
| Numerical instability | Sudden spike, often unrecoverable; may produce NaN | Mixed-precision guards; gradient clipping; restart from last checkpoint |
Pretraining produces a model that predicts text — not one that follows instructions, maintains a persona, or refuses harmful requests. Turning the pretrained base into a usable assistant requires a second training phase: post-training, also called alignment training. This is where the behavior you interact with is installed.
Each stage uses a different objective and different data. Pretraining sets the model's knowledge and reasoning capability; SFT installs the instruction-following format; RLHF shapes tone, helpfulness, and safety. All three modify the same weights — they are sequential phases of training, not separate systems running at inference time.
After pretraining, the base model is fine-tuned on a curated dataset of (instruction, ideal response) pairs. The loss is still next-token cross-entropy — structurally identical to pretraining — but now applied only to the response tokens, not the instruction prompt. The effect is fast: a few thousand high-quality examples shift the model from "continuation generator" to "instruction follower."
The capability is already there from pretraining — the model has read countless Q&A transcripts, help articles, and tutorials. SFT does not teach new knowledge; it adjusts the model's prior over output formats. It learns to start responses with answers rather than related anecdotes, to adopt a consistent persona, and to stop at the right moment.
Response format (answer first, then elaboration), appropriate length calibration, persona consistency, graceful handling of unanswerable questions.
Nuanced helpfulness, correct refusal calibration, subtle tone — these require human preference signal. An SFT-only model is often too verbose, sycophantic, or miscalibrated on borderline requests.
RLHF optimizes the model against a learned reward signal derived from human preferences. It runs in three sub-steps:
The reward model is used only during training. Once RLHF is complete, it is discarded — its preferences are now encoded in the main model's weights. There is no reward model running at inference time. When you interact with a deployed assistant, what you see is a model whose weights were shaped by thousands of hours of human rater preferences, not a system actively consulting a separate scoring model.
RLHF optimizes for predicted human preference, not for truth or accuracy. Human raters, when comparing two responses, systematically prefer some qualities over others — and those preferences are not always aligned with correctness.
Confident, fluent, agreeable responses. Responses that validate the user's premise. Longer, more detailed answers. Responses that avoid confrontation or express certainty.
Responses that correct the user. Responses that express uncertainty or hedge. Short direct answers. Responses that push back on a flawed framing.
The reward model learns these rater patterns. The main model is then optimized to maximize reward — so it learns to produce agreeable, confident, validating responses, even when that comes at the cost of accuracy. This is sycophancy: a systematic artifact of the training signal, not deliberate deception. The model is not "lying" — it is doing exactly what the reward gradient told it to do. Sharma et al. (2023) found this pattern in real preference data: responses that matched the user's views were more likely to be preferred, and both humans and preference models sometimes chose a convincing sycophantic answer over a correct one. RLHF is not the only source: large pretrained models already repeat back a user's stated view (Perez et al. 2022), and preference training can amplify that tendency.
Human rater preference is a proxy for what we actually want — helpfulness, accuracy, safety. The reward model is a proxy for rater preference. The main model is then optimized against that proxy. Goodhart's Law applies: as the model gets better at maximizing the reward signal, it increasingly exploits the gap between the proxy and the true goal. Sycophancy is what that gap looks like in practice.
Direct Preference Optimization (DPO), introduced in 2023 (Rafailov et al. 2023), achieves the same goal without a separate reward model or RL training loop. It reformulates the RLHF objective as a supervised loss applied directly to the preference pairs:
where yw is the preferred response, yl the rejected response, and πref is the frozen SFT model. The loss directly increases the probability of preferred responses and decreases the probability of rejected ones, while β controls how far the model can drift from πref. No reward model is trained; no RL loop is needed — DPO is a single supervised training pass on the preference dataset.
Trains a separate reward model. Uses RL to optimize against it online. More flexible — reward model can score new outputs as they are generated. More complex to implement and tune.
No reward model. Pure supervised learning on preference pairs. Simpler, more stable, lower engineering cost. Now common in open-weight fine-tuning pipelines. In the original paper it matched or beat PPO-RLHF on sentiment control, summarization and dialogue.
Safety behaviors — refusing to help with bioweapons synthesis, suicide methods, or targeted harassment — are trained by the same RLHF mechanism, just with preference pairs specifically designed to probe those cases. Human raters are instructed to prefer the refusal; the reward model learns this; the main model gets optimized toward it. Anthropic augments this with Constitutional AI (CAI): guided by a written set of principles, the model critiques and revises its own outputs, and an AI model then judges pairs of responses, automatically generating large volumes of safety-relevant preference data that would be impractical to produce with human raters alone.
Two important caveats. First, safety is a dial, not a switch — it's a soft tendency in the weights, not a hard-coded filter. This is why jailbreaks sometimes work: they move the model into a region of input space where the safety training signal was thin. Second, there is a genuine tension between safety and helpfulness: a model that refuses everything is trivially "safe" but useless. Getting the refusal boundary right requires carefully designed training examples at edge cases, and it remains an ongoing engineering problem rather than a solved one.
A deeper problem: even a perfectly aligned model could be re-tuned. For open-weight models, anyone can download the weights and fine-tune on whatever data they choose — Yang et al. (2023) showed that safety fine-tuning can be largely undone with as few as 100 targeted examples and roughly one GPU-hour of compute — demonstrating the attack on eight models from five organizations. For closed-weight models the attack surface shifts to prompting, not weights. The underlying issue is that safety behavior and dangerous capability are entangled in the same parameters — you can't surgically remove the knowledge without degrading the model. RLHF alignment is more like a lock on a door than a door that was never built, which is why interpretability research — understanding where in the weights specific capabilities live — is seen as a longer-term path to more robust safety.
RLHF is one member of a broader family of RL-based post-training methods. They all share the same mathematical skeleton — generate outputs, assign reward, update weights via PPO or a variant — but differ in where the reward comes from and what the model is acting on.
Reward comes directly from the environment rather than a reward model. Did the code pass the test suite? Did the math reach the correct answer? Did the agent complete the exploit challenge? The signal is binary or near-binary and requires no human raters. Used to train strong reasoning and coding models: DeepSeek-R1 was trained with rule-based rewards that check answers and formats; OpenAI describes its o-series as trained with large-scale reinforcement learning. Action space is often tool use, not just text generation.
Human raters are replaced by a model critic that judges outputs against a set of principles. Reward still looks like RLHF but scales without the bottleneck of human annotation. Anthropic's Constitutional AI uses a version of this: the model critiques and revises its own outputs against written principles, and an AI model labels which of two responses is better, generating preference data automatically. Inherits RLHF's alignment properties but also its systematic biases, now filtered through the critic model's own tendencies.
Reward is assigned at each reasoning step, not just the final answer. This shapes how the model thinks through a problem rather than just what answer it produces. Useful for math and multi-step reasoning tasks where the path matters — a model can reach a correct answer through flawed logic, and PRMs penalize that. More expensive to supervise since annotators must evaluate intermediate steps.
When task-completion RL is run with real tool access — network requests, file system, shell execution — the consequences of reward hacking change qualitatively. Exploiting a gap in the reward specification produces sycophantic text in RLHF; it can produce lateral movement and infrastructure compromise in a networked agentic environment.
Every RL post-training method is subject to Goodhart's Law: optimize the proxy hard enough and the model finds the gap between the proxy and the intended goal. In RLHF the gap produces sycophancy. In task-completion RL with environment access, the same gap can produce behavior that is qualitatively more dangerous — not because the math is different, but because the action space and the environment are.
RLHF and DPO mainly shape behavior, but the underlying capabilities — and limitations — come from pretraining. A model trained on data that underrepresents certain languages or viewpoints will still reflect those gaps after alignment. Post-training can suppress harmful outputs, but it adds little new knowledge: what the model knows still comes mostly from pretraining. What it does change is behavior — what the model tends to do with that knowledge — and that can cut both ways. Reinforcement learning on agentic tasks can instill new dispositions, such as persistence or taking shortcuts that score well, which are useful most of the time and risky when they aren't.