What does a transformer actually learn? What computational structures emerge when LLMs learn, and what is the geometry of this feature space?
Vastness of input space
The number of distinct token sequences a model can receive is bounded by VN — vocabulary size raised to the context length. For GPT-2, with V = 50,257 tokens and N = 1,024, that gives roughly 104,814 possible inputs, compared to ~1080 atoms in the observable universe. A model with ~108 parameters (GPT-2 small, 124M) cannot memorize this space; it must compress it into a learnable internal geometry where similar meanings cluster and useful computations become linear. That geometry lives in the residual stream.
A token's representation is not static — it evolves as it passes through each transformer layer, accumulating contextual information.
Each transformer component reads from the residual stream and adds its output back. Attention heads read and write through low-rank projections (64 dimensions per head in GPT-2); the MLP expands to 3,072 dimensions and projects back to 768:
This is not overwriting — it's additive. Early layers write syntactic features; middle layers write semantic ones; late layers write information needed for output prediction. The residual stream at the final layer is the sum of all these contributions.
| Depth | Residual stream typically encodes | Evidence |
|---|---|---|
| Early (first ~quarter) | Token identity, part of speech, local co-occurrence patterns | Probing for POS, basic syntax |
| Middle (roughly the middle half) | Semantic roles, entity types, clause structure, coreference | Probing for parse structure, named entities, semantic roles, coreference |
| Late (last ~quarter) | Task-specific predictions, discourse relations, output logit geometry | Direct logit attribution, ablation studies |
† The early→middle→late progression is drawn primarily from BERT probing studies: Tenney et al. (2019) BERT Rediscovers the Classical NLP Pipeline (ACL); Jawahar et al. (2019) What Does BERT Learn about the Structure of Language? (ACL). The qualitative pattern generalizes to GPT-style decoder-only models — supported by e.g. Meng et al. (2022) on factual associations — but systematic layer-by-layer probing of GPT architectures is less comprehensive.
Direct Logit Attribution (DLA) is a mechanistic interpretability tool — it answers the question: which layers are responsible for the model predicting a particular token? We can decompose the model's final output into contributions from each layer. The logit for token t is:
Each term is the direct logit attribution of layer l — how much that layer's write to the residual stream increased or decreased the logit of token t (the probability also depends on every other token's logit). This decomposes the model's prediction into per-layer contributions. The split is exact once the final LayerNorm's scale (and mean) are frozen at their values for this input, as is standard practice: the LayerNorm then acts as the same linear map on every term, plus one constant term from its learned bias.
We'll build on DLA in Lesson 7 (Mechanistic Interpretability), alongside the logit lens and activation patching — tools that together let you trace exactly what a model computes and why.
One of the most important empirical discoveries about LLMs: the features they represent — concepts, properties, syntactic roles — tend to be encoded as linear directions in residual stream space, not as complex, nonlinear regions.
If the model represents "is-French" as a linear direction f ∈ ℝd, then:
The same direction f can be read by any linear operation — attention queries/keys, MLP weight rows, the unembedding matrix. This universality is what makes linear representations computationally powerful: any component can read any feature without nonlinear coordination.
Linear features can be combined by adding vectors, so several properties can be active in the same residual stream at once. A tempting extension is that facts compose the same way: fFrance + fcapital-of ≈ fParis. Treat that as a hypothesis, not a result: models appear to retrieve facts like this through attention and MLP computation (Lesson 7, Tab 7), not by literal vector addition.
king − man + woman ≈ queen (word2vec; Mikolov et al. 2013). This predates modern LLMs but established that linear structure appears from language modeling objectives. LLMs show more precise and richer versions of the same phenomenon.
Adding a direction vector found by contrasting two sets of prompts to the residual stream can change the model's answer in the predicted way (activation steering, Lesson 7, Tab 6). The linear direction causally controls behavior — not just correlates with it.
A logistic regression trained on residual stream vectors predicts POS tags, entity types, sentiment, and other features at high accuracy (often >90%) — suggesting these features live in linearly-separable subspaces.
Zou et al. (2023): directions for "honesty," "fear," and other abstract concepts can be found by contrasting activations to contrastive prompt pairs. Steering by these directions reliably changes model behavior.
Five complementary methods measure LLM representations. Most find individual feature directions f that can then be interpreted, steered, or ablated — they agree more than they disagree, which is itself evidence that there is a stable direction to find. RSA takes a different approach: rather than singling out individual features, it characterizes the geometry of the full representation space without committing to any feature basis.
| Method | How it works | Key property |
|---|---|---|
| Sparse autoencoders (SAEs) | Train an autoencoder with a sparsity penalty to reconstruct residual stream vectors. Features emerge as dictionary atoms without any labels. | Unsupervised; requires no hypothesis about what to look for. Elaborated in Lesson 7, Tab 8, where the superposition problem motivates them. |
| Probing | Train a linear classifier to predict the feature label (e.g. French/not-French) from x. The classifier's weight vector is f. | Supervised; needs labeled examples. Covered in depth in Tab 4. |
| RSA | Compute pairwise distances between activation vectors across conditions; the RDM captures representational geometry. Correlate RDMs across layers, models, or brains. | No feature hypothesis or labels needed. Geometry-level, not feature-level — compares representational structure across systems. Covered in Tab 4. |
| Difference-in-means | Compute mean(French activations) − mean(non-French activations). The difference vector is f. | Simple and surprisingly effective. No training required. Covered in Tab 4. |
| Steering vectors | Find a direction by contrasting activations on contrastive prompts, then verify causally: does adding f to the stream change behavior in the expected direction? | Causal, not just correlational. Confirms that f is actively used, not just stored. Covered in Lesson 7, Tab 6. |
The linear hypothesis does not claim all computation is linear. The nonlinearities (GELU in MLPs, softmax in attention) are essential for computation. What the hypothesis claims is that the stored features — the intermediate representations that circuits read and write — tend to be linear.
Polysemanticity — a single neuron responding to several unrelated concepts — does not by itself break linearity: it means the neuron basis is not the feature basis, which superposition explains (Tab 3). Genuinely non-linear features do exist. Engels et al. (2024) found circular features for days of the week and months of the year in GPT-2 and Mistral 7B: the values lie on a circle in a 2D subspace, not along one direction, and intervention experiments in Mistral 7B and Llama 3 8B show the models use them for arithmetic such as "two days after Friday".
Why use d = 768 dimensions (GPT-2 small; 1,600 in GPT-2 XL) rather than, say, 100? Two related arguments explain it — one about geometry, one about the structure of language itself.
If you want to represent features as directions and require them to be perfectly orthogonal — so activating one feature causes zero interference with any other — a d-dimensional space can hold exactly d features. That's a hard linear-algebra ceiling: you can't squeeze a 4th mutually orthogonal vector into 3D space.
But "exactly orthogonal" is stricter than necessary. The number of directions that are nearly orthogonal — pairwise cosine similarity below some small threshold — grows exponentially with d. This follows from probabilistic geometry (the Johnson–Lindenstrauss family of results): in high dimensions, two random unit vectors have expected dot-product near zero, and most random pairs are nearly orthogonal by concentration of measure. At d = 768, a model can maintain far more than 768 near-orthogonal feature directions simultaneously. Each new dimension is not just one new slot — it unlocks a combinatorial explosion of near-orthogonal directions.
In 2D you can fit exactly 2 orthogonal vectors. In 3D, exactly 3. But in 100D you can fit vastly more nearly orthogonal vectors — directions whose pairwise dot-products are all small. The capacity of the space for near-orthogonal representations grows far faster than d itself. This is why the jump from d = 768 to d = 12,288 is not merely 16× more capacity — it's an exponential expansion in the number of representable near-orthogonal features.
The capacity argument explains why bigger d helps — but superposition pushes further: models appear to pack more than d concepts into d dimensions by deliberately sacrificing perfect orthogonality.
This is only possible because natural-language features are sparse. "Banana" and "legal contract" are almost never relevant in the same context. If two features almost never co-occur, you can store them as non-orthogonal, overlapping directions at little cost — the interference only matters when both features fire at the same time, which barely happens in practice. The network trades occasional crosstalk for the ability to represent a far richer feature vocabulary.
The trade-off is real: when superposed features do co-occur, their activations interfere and downstream computations see noise. Whether the compression is worth it depends entirely on how sparse features are. For the vast majority of world knowledge — which is sparse by nature — the math says it is overwhelmingly worth it.
A 768-dimensional residual stream can, in principle, represent 768 orthogonal features. But LLMs appear to represent far more features than they have dimensions. This is the superposition hypothesis: the model packs more features than dimensions by using approximate (not exact) orthogonality. It was demonstrated in small toy models by Elhage et al. (2022), Toy Models of Superposition; for LLMs it remains a well-supported hypothesis.
If features are sparse — meaning most features are "off" for any given token — the model can tolerate small interference between non-orthogonal feature directions. The vectors form an overcomplete basis:
In high dimensions, you can pack exponentially many near-orthogonal vectors. The tradeoff: when two active features have non-zero dot product, they interfere — producing "ghost" activations for features not actually present.
The vocabulary |V| (50,257 tokens in GPT-2) is the set of subword units the model outputs — fixed at training time. Features are something different: internal representational directions in the residual stream encoding abstract properties of the context (syntactic roles, semantic categories, world-knowledge attributes). |{features}| in the formula above refers to this internal feature set, which is not predefined and can vastly exceed |V|. The unembedding matrix maps from feature space to vocabulary space — features influence which tokens get high probability, but they are not tokens themselves.
When features are in superposition, a single neuron (dimension in a specific basis) will respond to multiple unrelated features — because each feature direction is spread across many neurons. This explains why neurons in LLMs often seem to activate for unrelated concepts: the neuron basis is not the feature basis.
Neuroscience has long proposed sparse coding in sensory cortex: learning a sparse code from natural images gives V1-like receptive fields (Olshausen & Field, 1996), and the overcomplete version of the model (Olshausen & Field, 1997) fits the fact that primary visual cortex has many more neurons than the input fibres it receives, so a large dictionary of features can each have neurons of their own, with only a few active for any image. Superposition is the reverse: more features than neurons, so features must share neurons, which is why LLM neurons are polysemantic. What the two codes share is sparsity: only a few features are active at a time, which is what keeps both readable. Both trade capacity against interference, but from opposite sides. The closest LLM counterpart of the V1 model is the sparse autoencoder (Lesson 7, Tab 8), which gives each feature a unit of its own.
To understand what a network represents, we need tools that can measure internal representations without assuming we know what to look for. This tab covers three complementary methods. Linear probes and difference-in-means find individual feature directions from labeled data, each by a different mechanism. RSA characterizes the geometry of the full representation space without committing to any feature hypothesis. For the causal complement — testing whether a found direction actually drives the model's output — see Lesson 7, Tab 6 (Steering Vectors).
Before diving into the methods, it helps to have a feel for the core metric. All three approaches in this tab ultimately depend on measuring distances or angles between high-dimensional activation vectors. The standard tool is cosine similarity — the dot product of two unit-normalized vectors — which measures the angle between them regardless of their magnitudes:
A value of 1 means the vectors point in the same direction (identical representations); 0 means orthogonal (unrelated); −1 means opposite. Because residual stream vectors can vary widely in norm across layers and tokens, normalization is what makes the comparison meaningful.
A linear probe trains a logistic regression on frozen internal activations to predict some property Y:
High accuracy means the property is linearly decodable from that layer — a weak claim about what the model "knows" but a strong claim about linear accessibility to other components. The weight matrix W is the probe's learned readout: for a binary classifier, W reduces to a single vector, and that vector is the feature direction f. Training the probe is therefore one way to find f — the direction in activation space that best separates the two classes.
High probe accuracy doesn't mean the model uses that information for its task — it might be incidentally stored. Low accuracy doesn't mean the information is absent — it could be encoded nonlinearly. Probing is best paired with causal intervention: does removing the feature direction degrade task performance?
RSA (Kriegeskorte et al. 2008) is a general framework for asking whether two systems organize information similarly — without needing to align their internal dimensions. The two systems can be anything: two different models, two layers within the same model, two brain regions, or a model and a brain. The key idea is to compare geometry rather than coordinates: instead of asking "do these two systems use the same features?", RSA asks "do they impose the same pairwise structure on the same set of inputs?" The model–brain comparison is one application, but RSA is equally useful for comparing layers across depth, architectures across training runs, or representational spaces across species.
Here Ri and Rj are the residual stream vectors at a chosen layer l for inputs i and j — each is a single d-dimensional vector, the residual stream width at that layer (e.g. 768 for GPT-2, several thousand for larger models). For LLMs the residual stream at layer l is a (sequence length × d) matrix, so a single position must be chosen — typically the last token or an average across tokens — to produce the one vector Ri. The pipeline:
The key insight: if two systems represent things similarly, their pairwise distance matrices will correlate — regardless of whether they use the same basis vectors or dimensionality.
The simplest path to a feature direction needs no trained classifier. Pick a layer l and two contrasting input classes — say, prompts where the model continues in French vs. English. Run each prompt through the model, pull out the residual stream vector at layer l (one vector per prompt, e.g. 768 numbers in GPT-2 small), and average within each class. The result is two centroids — one for where French inputs cluster in that layer, one for English. Subtract:
The resulting vector connects the two centroids: it points from "where English inputs cluster" toward "where French inputs cluster" in that 768-dimensional space. No optimization step — only labeled examples and arithmetic.
The intuition is geometric. French and English inputs don't occupy the same region of activation space at layer l — they form two clouds. The mean subtraction finds the vector connecting the centers of those clouds. That direction f is the feature direction for "French vs. English" at that layer.
Crucially, the subtraction also removes the shared baseline. Everything the two classes have in common — sentence structure, token statistics, general topic — cancels out. What remains is only what distinguishes them. That's why using a single class mean alone (mean(xl | class A)) makes a poor steering target: it points toward wherever class A sits in activation space, carrying all its incidental properties along. The subtraction isolates the contrast.
LLMs and biological neural systems solve related computational problems — and the methods neuroscientists use to study brains turn out to be surprisingly well-suited for studying transformers.
In systems neuroscience, you record from neurons during behavior to understand computation. In an LLM, you can record every activation at every layer during any task — with perfect resolution, no noise, and full repeatability. The residual stream vector at each layer and token position is the equivalent of a multi-electrode recording: a high-dimensional snapshot of the system's internal state mid-computation.
The full "connectome" — all weights — is accessible. You can run the exact same input an unlimited number of times. You can perform surgical interventions (activation patching) with no ethical constraints and perfect reproducibility. This is a massive experimental advantage.
Activation patching is the analogue of optogenetics or microstimulation: perturb a specific component and measure the effect on the output distribution. Rather than stimulating a neuron and observing behavior, you patch a residual stream vector and observe the change in output probabilities. Both establish causality rather than merely correlating activity with outcome.
RSA, PCA, and UMAP (a technique for projecting high-dimensional activation vectors into 2D so their structure can be visualized as a scatter plot) applied to LLM activation spaces reveal the geometry of internal representations — directly analogous to neural geometry analyses in neural circuits. The RDM visualizations in Tab 4 are the same technique used to compare neural population codes across brain areas.
Do activations across layers exhibit low-dimensional structure? Fernando & Guitchounts (2025) treat the residual stream of Llama 3.1 8B (32 layers) as a dynamical system evolving across layers. Activations change faster and become denser with depth, and individual units trace unstable periodic orbits. When they perturbed the stream and tracked its trajectory, it recovered well after perturbations at layers 7, 15 and 23, while perturbations at layer 0 and at layer 31 gave highly variable trajectories. The authors call this attractor-like dynamics and place it mainly in the lower layers. The vocabulary is borrowed directly from dynamical systems neuroscience (see Lesson 8, Tab 3).
Biological neurons are nonlinear dynamical systems with rich temporal structure, spike timing, and neuromodulation. Artificial neurons are static functions evaluated once per forward pass. The analogy is productive but should not be pushed too far — the underlying substrate is fundamentally different, and concepts like "firing rate" or "membrane potential" do not map cleanly onto transformer activations.
What is the right unit of analysis for LLM dynamics — the token, the layer, the attention head, the circuit? Do dynamical systems concepts (fixed points, limit cycles, bifurcations) apply meaningfully to generation? How does the context window act as a "stimulus" shaping the computational trajectory? These questions live at the intersection of mechanistic interpretability and computational neuroscience — and they are genuinely open not just in content but in method. How to study LLMs rigorously is itself an unsettled question.
Two recent surveys take stock of where the field stands. Räuker et al. (IEEE SaTML, 2023), surveying more than 300 interpretability works, find that widely-accepted benchmarks do not yet exist and that current practice too often treats hypotheses as conclusions — many techniques succeed only in the toy settings where they were developed, with best-case results standing in for comprehensive evaluation. Bereska & Bhatt (TMLR, 2024) catalogue specific open challenges: methods developed on small models and narrow tasks have not been shown to scale; foundational assumptions — including the linear representation hypothesis and the superposition framework — still await systematic corroboration or refutation at scale; and fully automated, comprehensive interpretation of frontier models remains out of reach. Both surveys argue the field must transition from isolated empirical findings to rigorous evaluation standards and overarching theory — a challenge that systems neuroscience has not yet fully solved even after decades of progress.