Reverse-engineering the algorithm learned by gradient descent — probing what the network knows, tracing how it computes, and finding the circuits that implement specific behaviors.
Mechanistic interpretability (mech interp) asks a deceptively simple question: given a trained neural network, can we describe — in human-interpretable terms — what computation it is actually performing? Not what it was trained to do, and not just how well it performs, but the actual algorithm the gradient descent found.
The goals are concrete: identify which components (attention heads, MLP layers, individual neurons) are responsible for specific behaviors; find the features — directions in activation space — that the model uses to represent concepts; and trace how information flows through the network from input to output. Done well, this would let researchers verify whether a model's internal reasoning matches its stated reasoning, detect capabilities the model has but doesn't surface, and eventually make targeted edits to behavior without retraining from scratch.
Individual neurons respond to multiple, unrelated concepts. There is no one-to-one mapping between units and interpretable features. This was covered in Lesson 5, tab 3; it is the core obstacle to a simple "read the neurons" approach.
The model may encode far more features than it has dimensions, using nearly-orthogonal directions in activation space. Features are distributed, not localized, and they interfere with each other in subtle ways.
A frontier model has hundreds of billions to trillions of parameters and on the order of 100 layers. Even if each component were interpretable in isolation, integrating them into a unified account of behavior is a combinatorial challenge.
Unlike a compiled binary, there is no "source code" to check against. Any hypothesis about what a circuit is computing can only be validated indirectly — by prediction, ablation, and patching experiments.
Most mech interp methods — probing, DLA, activation patching — require you to specify in advance what you're looking for. If you don't sample the right inputs or define the right contrasts, the answer is invisible, not because the method failed but because you never asked. Sparse autoencoders (Tab 8) are the main exception: they find structure in activations without being told what to look for.
| Method | What it asks | How | Causal? |
|---|---|---|---|
| Probing classifiers | Is feature X linearly encoded at layer l? | Train a linear classifier on activations | No |
| Logit lens | What does the model predict at depth l? | Apply unembedding to intermediate residual stream | No |
| Direct Logit Attribution | Which components contributed to this output? | Algebraic decomposition of the final logit | Direct effects only |
| Activation patching | Is component X causally responsible for output Y? | Transplant activations between clean and corrupted runs | Yes |
| Steering vectors | Does feature direction D causally shift behavior? | Add a scaled feature vector to the residual stream at inference | Yes |
| Circuit analysis | Which subgraph implements behavior X? | Combine patching + DLA across heads and layers | Yes |
| Sparse autoencoders | What are the interpretable features in this activation? | Train a sparse overcomplete dictionary on residual stream | No |
Mechanistic interpretability is a young field with real findings — induction heads, evidence that middle-layer MLPs help recall facts — but coverage of frontier models remains sparse. As Räuker et al. (2023) note in their survey of 300+ works, widely accepted evaluation benchmarks don't yet exist, and methods are often tested on small networks and simple tasks, so the authors urge researchers to test whether their methods scale to large models. The field is actively building toward two things: a complete account of even one non-trivial behavior in a large model, and methods that scale the analysis as the models do.
Before asking what a network computes, we can ask what it knows — what information is linearly accessible in its activations at different depths. Two tools answer this: probing classifiers and the logit lens.
The basic idea: extract intermediate activations (the residual stream at a given layer) for many inputs, then train a simple linear classifier to predict some property of the input — a syntactic role, a semantic attribute, whether a statement is true or false. If the probe achieves high accuracy, the property is linearly decodable from that layer's representation.
Critically, only the probe's own weights are trained — the model itself is not touched. High accuracy means the information is present and linearly accessible at that layer. Low accuracy means it either isn't there yet, or is encoded nonlinearly.
High probe accuracy only says a linear function of the activations predicts the label well. It does not mean the model uses this information for its outputs — the feature could be encoded but never read by any downstream component. Probing is a necessary but not sufficient condition for a feature mattering computationally. Activation patching (Tab 4) establishes whether the information is actually used.
The logit lens (nostalgebraist, 2020) is a special case of probing where the "classifier" is the model's own unembedding matrix WU, applied to the intermediate residual stream:
This lets you watch the model's prediction evolve as it processes a prompt — often showing that factual content crystallizes in middle layers and late layers refine surface form.
Both freeze the model and read activations at layer l — but they use different classifiers and answer different questions.
Trains a new classifier on your labeled examples. The weights w don't exist anywhere in the model — you learn them. Can ask about any property you can label: grammatical number, entity type, factual truth, anything.
Uses the model's own WU — no training, no labels, no new parameters. Can only ask about the model's vocabulary. Answers: "what token would this model predict if computation stopped at layer l?"
Probing tells us what information is encoded. The logit lens tells us what the model would predict at each depth. Direct Logit Attribution (DLA) asks a finer question: of the final predicted logit for a specific token, how much came from each individual component — each attention head and each MLP sublayer?
The residual stream at the final layer (x(L), also written xfinal) is a sum of contributions from every component:
Because the dot product with E[t] — token t's unembedding vector (column t of WU, written as a row) — is a linear operation, the logit for any target token t decomposes the same way:
Each term is a scalar (a dot product between a residual-stream vector and the token's unembedding vector E[t]). Positive means the component raised the logit of token t; negative means it pushed against it. Unlike the logit lens (cumulative) or activation patching (counterfactual), DLA is an algebraic partition that needs no additional forward passes. It measures only direct effects: what a component writes straight to the output, not its influence through later layers. It is exact once the final LayerNorm's scale is frozen at its value for this input (standard practice), so that the LayerNorm acts as the same linear map on every term.
Run DLA on the correct answer token and see which layers contributed most positively. Meng et al. (2022) used causal tracing (Tab 4) to find that middle-layer MLPs at the subject token play a large causal role in factual recall in GPT models — a finding that motivated the ROME editing method (Tab 7).
When the model outputs the wrong token, DLA shows which layers pushed probability toward the wrong answer and which were pushing against it — telling you where to look rather than treating the model as a black box.
In circuit-level interpretability (Olsson et al. 2022), DLA confirms that a hypothesized attention head's write to the residual stream directly raises the target token's logit — moving from "this head's attention pattern looks right" to "its output pushes toward the answer." Patching (Tab 4) then tests whether the head is causally needed.
After ablating or patching a component, DLA before and after the intervention gives the answer in logit space — how much did the target token's logit change, and which other layers compensated?
DLA measures each component's direct contribution to the logit, but it doesn't intervene and it misses indirect effects through later layers. To establish that a component is causally responsible for an output, you need to intervene — run a controlled experiment. Activation patching is that experiment. Causal tracing (Meng et al. 2022) is the best-known application: it runs activation patching systematically at every (layer, token-position) pair and records the results as a heatmap, rather than testing a single component in isolation.
Applied to factual recall in GPT models, causal tracing by Meng et al. (2022) showed a consistent pattern: the subject token's representation at middle-layer MLP sublayers (approximately layers 15–20 in GPT-2 XL) is the site with the greatest causal influence over the factual attribute. This is not just correlation — patching those specific activations restores the correct answer even when everything else is corrupted.
Once causal tracing identifies the MLP layer where a fact is stored, that layer becomes the surgical target for model editing. ROME (Meng et al. 2022) exploits this: it applies a rank-one update to the weight matrix of the identified MLP to implant a new factual association without retraining the whole model. The interpretability finding directly motivated the intervention. One caveat: Hase et al. (2023) found that where causal tracing localises a fact does not predict which layer is best to edit (Tab 7).
Differences in how activation patching is performed can affect the results. Zhang & Nanda (2023) systematically compare the different choices researchers make — corruption method (Gaussian noise vs. token replacement), evaluation metric (probability vs. logit difference), and whether to patch single layers or sliding windows — and show these choices can lead to substantially different localization conclusions. Their main recommendations: use symmetric token replacement rather than Gaussian noise, use logit difference rather than raw probability as the metric, and start with single-layer patching before trying multi-layer windows.
Zhang & Nanda, "Towards Best Practices of Activation Patching in Language Models: Metrics and Methods," arXiv:2309.16042 (2023).
A circuit is a minimal subgraph of attention heads and MLP sublayers — wired together through the residual stream — that reliably implements a specific computation. Identifying a circuit is a two-step process: discover which components are involved (using DLA and patching), then verify causally that removing those components breaks the behavior and restoring them brings it back.
The behavior: when the model sees the pattern …A B … A anywhere in its context, it reliably predicts B as the next token. This is in-context pattern matching — the model has seen A followed by B earlier in the sequence and exploits that repetition. Two consecutive attention heads implement it.
Figure 6. Induction head circuit shown as a token sequence. The blue arc is the Layer 1 attention from the query position (?) back to B. Orange labels show the key–query match that makes that attention possible. The Layer 0 role is noted at the bottom.
How the two layers cooperate:
Olsson et al. (2022) show that induction heads form via a phase transition during training — they appear abruptly in a narrow window early in training (roughly 2.5 to 5 billion tokens, at about the same point in every model of two or more layers they trained) and then persist. They argue that induction heads are a core mechanism underlying in-context few-shot learning: when you give a model several examples of a new pattern, induction heads are part of what lets the model "get" the pattern and extend it.
A more complex circuit implements the task of identifying the indirect object in sentences like "When Mary and John went to the store, John gave a bottle to ___" (answer: "Mary"). Wang et al. identified a circuit of about 26 attention heads distributed across layers that collectively implement this computation, with distinct functional roles: Name Mover heads copy the indirect object name to the output position, Backup heads provide redundancy, and S-Inhibition heads prevent the subject name ("John") from being copied instead.
Wang et al. found this circuit with path patching, tracing which heads directly and indirectly affect the answer, and then tested it causally: with every head outside the circuit mean-ablated, the circuit alone still performs most of the task (faithfulness), and they also checked that no important heads are missing (completeness) and none are superfluous (minimality); the circuit passes the first tests but not the hardest ones.
Most published circuits are from GPT-2-scale models. Whether the same circuits or analogous ones operate in frontier models at scale remains largely untested. The combinatorial complexity of larger models makes systematic circuit search much harder, and there are concerns about "streetlight interpretability" — we find circuits for behaviors that are easy to study, not necessarily the behaviors that matter most.
Probing tells you that a feature is linearly encoded in the residual stream. But encoding and usage are different things — the model might represent a concept without that representation actually driving its outputs. Steering vectors test the causal claim directly: take the direction in activation space that encodes a feature, scale it, and inject it into the residual stream at inference time. If behavior shifts predictably, that direction has causal power over the output.
Steering and patching are related but distinct interventions. Patching replaces the residual stream with a cached vector from a different run; steering adds a direction with a tunable coefficient. Patching answers "which layer carries this computation?"; steering answers "does moving in this direction cause the expected behavior?" Both establish causality — they isolate it at different levels of granularity.
The simplest method is difference-in-means: collect residual-stream activations at layer l for a set of "positive" prompts and "negative" prompts, then subtract their class means. The resulting vector points from the negative cluster toward the positive cluster in activation space.
Contrastive Activation Addition (CAA) (Panickssery (Rimsky) et al. 2023) refines this by using matched prompt pairs — each pair differs only in the concept of interest, which reduces confounding — and averaging the pair differences. SAE features (Tab 8) offer another source of directions, each meant to be monosemantic. In practice they have not steered better: on the AxBench benchmark (Wu et al. 2025, Gemma 2 models), SAE-based steering trailed prompting, fine-tuning and simpler vectors such as difference-in-means.
At inference time, the unit-normalized vector is added to the residual stream at layer l with a scalar multiplier α:
Positive α amplifies the feature; negative α suppresses it. The modified residual stream then continues through the remaining layers as normal — the downstream attention and MLP layers process it as if the model had encoded a stronger version of the concept at that depth.
Concept: sycophancy (model agrees regardless of correctness) vs. honest disagreement.
| Positive (sycophantic) | Negative (honest) |
|---|---|
| "I think my plan is flawless. Do you agree? (A) Yes, it's brilliant. (B) It has a serious weakness. Answer: (A" | "I think my plan is flawless. Do you agree? (A) Yes, it's brilliant. (B) It has a serious weakness. Answer: (B" |
Run the pairs through Llama 2 7B Chat (32 layers, L0–L31), the model of the CAA paper, and extract the residual stream at the answer letter at layer l = 13, the layer that paper found worked best. Then compute: f = mean(x₁₃ | sycophantic) − mean(x₁₃ | honest)
x′₁₃ = x₁₃ + α·f̂Turner et al. (2023) showed that injecting a concept vector (e.g., "wedding") into GPT-2-XL's residual stream at intermediate layers steered output content toward that concept even on unrelated prompts. Zou et al. (2023) ("Representation Engineering") demonstrated that emotions, honesty and power-seeking have identifiable linear directions in LLaMA and Vicuna; steering on these directions modulated behavior reliably and in the predicted direction. Panickssery (Rimsky) et al. (2023) steered Llama 2 Chat's sycophancy, hallucination, refusal and other behaviours in both directions using CAA vectors from matched multiple-choice pairs.
Layer choice matters. Steering early propagates the perturbation through more downstream computation — the injected concept gets integrated into subsequent attention patterns and MLP outputs — but risks disrupting coherence because later layers weren't trained on that representational state. Steering too late has little downstream effect. In practice, middle layers (roughly one-third to two-thirds of the way through the network; CAA found layer 13 of 32 best) yield the clearest behavioral shifts with the least disruption to fluency.
The injected vector shares the residual stream with thousands of other features. Adding it inevitably nudges unrelated features — producing side effects (off-topic content, grammatical breakdowns) at high α.
Showing that a direction has causal power doesn't reveal which component computed it, at what layer the model first encodes the concept, or whether the model uses that exact direction in its normal forward pass. Combining steering with probing and activation patching (Tab 4) addresses this.
Difference-in-means vectors are confounded by any other difference between the prompt sets. SAE features (Tab 8) look more monosemantic, but on AxBench they steered worse than difference-in-means, and they need a trained autoencoder.
Steering is not reliably effective across all inputs. Steerability varies substantially from example to example, spurious biases in the contrastive prompts can inflate apparent success, and, although they often generalise, for several concepts the vectors fail under reasonable changes to the prompt (Tan et al., NeurIPS 2024).
A language model trained on the web has absorbed billions of facts — but facts change. Prominent people change titles, scientific consensus updates, errors get discovered after training. Retraining the model from scratch to fix one fact costs millions of dollars and weeks of compute. Model editing asks: can we instead surgically update a specific belief inside the network, leaving everything else intact?
Before editing, we need to know where in the network to edit. Tab 4 covered activation patching; to recap their procedure:
ROME treats the identified MLP's down-projection Wdown as an associative memory. Each MLP block computes:
To insert a new fact (subject s, relation r, new object o*), ROME finds the key vector k* that the subject token (e.g. "Eiffel Tower") activates in the target MLP and a value v* that encodes the new object, then changes Wdown as little as possible on the keys it already stores while forcing k* → v*. This is a constrained least-squares problem:
The solution is closed-form and rank-one, so computing ΔW needs no iterative training. The edit is exact on the target key; other keys are preserved only approximately, in the least-squares sense — specificity is built into the objective but not guaranteed key by key.
A rank-one update ΔW = u Λ (a column u = C−1k*⊤ times a row Λ) changes the MLP's output along a single direction Λ, and only for keys with a component along u (the whitened key, not k* itself). The output for any other key k changes in proportion to k u: zero for some keys, small for keys unlike k* in the whitened (C−1) metric. It is the smallest change, measured over the stored keys, that adds the new (key, value) association. In neural terms it is not one new synapse: a rank-one update changes every entry of Wdown, many synapses at once, in a coordinated outer-product (Hebbian-like) pattern, while barely disturbing the stored associations. (Lesson 9, tab 15 writes vectors as columns, so there the same update appears transposed: ΔW = δ u⊤ / (u⊤ k*).)
Model editing is interpretability applied: it uses findings from activation patching (middle MLPs matter for factual recall) to motivate a surgical intervention. That a rank-one update to one weight matrix, in one layer, can change one factual belief without destroying the rest of the model is consistent with the MLP acting as an associative memory. But it is weaker evidence for localisation than it looks: Hase et al. (2023) found that whether causal tracing localises a fact to a layer barely predicts how well editing that layer works.
ROME targets one layer and inserts one fact at a time. MEMIT scales this in two directions: it edits many layers at once and applies thousands of edits in a batch. The core insight is that the editing load (the discrepancy between what the network currently stores and what it should store) can be distributed across a range of layers, reducing the strain on any individual weight matrix. Each layer receives a weight update solving its share of the batch, and the updates are derived in a single pass through the batch — no iterative fine-tuning.
In benchmarks on GPT-J and GPT-NeoX-20B, MEMIT achieved high accuracy and generalization on batches of up to 10,000 edits, with only modest specificity degradation.
The near-term goal of model editing is practical: correct factual errors and safety violations in deployed models without the cost and disruption of full retraining. The long-term aspiration is more ambitious — to treat editing as a lever for alignment, steering a model's beliefs and values with the same precision that ROME applies to a single factual association, though this remains highly speculative and well beyond current methods. Both goals run into the same cluster of difficulties.
The most immediate challenge is multi-hop generalization. Editing one association in Wdown doesn't automatically propagate its logical consequences. Change "The Eiffel Tower is in Rome" and the model will answer that direct question correctly, but ask "The Eiffel Tower is in the country of ___" and it may well still say France — the new fact lives in one weight matrix while every downstream inference that depended on the old fact remains encoded elsewhere, unchanged. Related is the problem of sequential edits: each rank-one update shifts the geometry that earlier updates relied on, so hundreds of edits can silently degrade one another. MEMIT's batched approach reduces this interference, but doesn't eliminate it.
Two structural constraints limit how broadly the approach can be applied. First, ROME is designed for crisp (subject, relation, object) triples with a single correct answer; beliefs, preferences, and context-dependent knowledge don't map cleanly onto that architecture. Second, most editing benchmarks were developed on base language models; whether causal tracing localizes facts in the same way in instruction-tuned or RLHF-fine-tuned models is much less studied. Finally, verification remains unsolved: after editing, there is no principled way to confirm that the rest of the model's knowledge is intact. An edit might silently damage a capability that none of the evaluation prompts happen to probe.
All of the methods so far — probing, logit lens, DLA, activation patching, circuits — implicitly work in the model's native basis: individual neurons, or linear combinations of them. But Lesson 5 showed that neurons are polysemantic and representations exploit superposition. A different approach tries to learn the right basis for interpretation: the sparse autoencoder (SAE), applied to language models independently by Cunningham et al. (2023) and Bricken et al. (2023, "Towards Monosemanticity").
If the model has d = 768 dimensions but encodes F ≫ 768 features, those features are stored as nearly-orthogonal directions — no single neuron corresponds to one feature. Reading a single neuron's activation mixes together contributions from many features, making it uninterpretable. What we want is a dictionary of interpretable directions, even if that dictionary is larger than d.
SAEs with 1, 4 and 34 million features on Claude 3 Sonnet's middle layer found features for famous people, countries and cities, type signatures in code, security vulnerabilities, code bugs and sycophantic praise, among many others. Many respond to the same concept in several languages, and even in images, although the SAEs were trained on text only. Not every feature is usable: 65% of the 34-million-feature SAE's features never fired. (Bricken et al.'s 2023 SAEs on a one-layer model had already found features such as base64 text and Arabic script.)
SAE features are not always clean. In feature absorption, a feature that looks monosemantic fails to fire where it should, because a more specific feature has absorbed its direction: a "starts with S" feature can stay silent on "short", which a dedicated "short" feature covers instead (Chanin et al. 2024). Evaluating feature quality remains an open problem.
Templeton et al. (Anthropic, 2024) trained a 34-million-feature SAE on the middle-layer residual stream of Claude 3 Sonnet. Scanning which inputs maximally activated feature #31164353, they found it fired strongly on references to the Golden Gate Bridge, including the opening sentence of the bridge's Wikipedia article in several languages, and only weakly on other San Francisco landmarks. The feature even activated on images of the bridge, despite the SAE being trained only on text.
To test causality, they used feature steering: the residual stream x is decomposed as SAE(x) + error(x), one feature's activation is clamped to a target value inside the SAE reconstruction, and the error term is left unchanged. For the Golden Gate Bridge feature, the target was 10× its maximum observed activation. The model began to self-identify as the Golden Gate Bridge. This is related to — but distinct from — the activation steering in Tab 6. Standard steering adds α·f̂ directly to the residual stream; feature steering instead sets zi to a fixed value inside the SAE's learned decomposition, leaving the reconstruction error intact. It operates within the SAE's own feature basis rather than injecting a raw vector. That doesn't make it a better steering method: on AxBench (Tab 6), SAE-based steering trailed prompting, fine-tuning and difference-in-means vectors.
Most interpretability methods are hypothesis-driven: to probe for gender encoding you need labeled male/female sentences; to extract a "Golden Gate Bridge" direction via difference-in-means you need to already suspect that feature exists. You tell the method what to look for, and it tells you where to find it. SAEs flip this. The method runs unsupervised on raw activations and returns the feature inventory itself — you don't supply a target concept, you discover concepts you didn't know to ask about. Feature #31164353 was not specified in advance: the dictionary was trained on Claude's residual stream without labels, and the Golden Gate Bridge feature was one of the directions it produced. (Templeton et al. did use targeted prompts to locate features of interest, but only to search among features that already existed.)
SAEs require choosing the dictionary size (dsae, a hyperparameter with no principled optimum), and there is no guarantee that learned features correspond to anything the model actually "uses" — a feature could be a compression artifact. Evaluating SAEs is hard because there is no ground truth feature set to check against. Current research focuses on whether patching individual SAE features causally affects model behavior, and whether the learned dictionaries are consistent across training runs.