Lesson 8 · Neural Systems

The Fundamental Analogy

Neuroscience spent much of the 20th century trying to understand how populations of neurons represent and transform information. The core empirical tool was the electrode: place a probe near neurons, record their voltage spikes, and try to infer what the population is "computing." Decades of progress followed — from simple receptive fields in V1 to population vectors in motor cortex to place cells in hippocampus.

LLMs offer something stranger and more tractable: a neural network with the same functional challenge (mapping inputs to outputs via learned representations), but with full observability. We can record from every "neuron" at every "timestep." We can replay stimuli exactly. We can intervene with surgical precision. We can examine the weights that produced the behavior. In many ways, an LLM is the dream experiment that biological neuroscience never gets to run.

The core analogy: Tokens in an LLM correspond roughly to stimuli. Layers correspond to time (or to the depth of processing). The d dimensions of the residual stream are most analogous to neurons. The residual stream at each layer is the population state — a high-dimensional vector summarizing "what the network knows" at that moment. Studying how this state evolves from layer to layer is, structurally, very similar to studying how neural population activity evolves over time during a task. For autoregressive generation, each forward pass is analogous to a trial dependent on the previous trial history. While these analogies are imperfect, we will explore if any concepts from electrophysiology are potentially useful for studying LLMs.

What "recording" looks like in an LLM

In classical electrophysiology, you insert an electrode and record a spike train — a sequence of action potentials over time. In a modern large-scale recording experiment (e.g., silicon probes), you might record from hundreds or thousands of neurons simultaneously, capturing a population trajectory through high-dimensional state space as the animal performs a task.

In an LLM, the analog is to extract the residual stream activations at each layer. For a model with d dimensions and L layers processing a sequence of T input tokens, you obtain a tensor of shape T × L × d. Each token's trajectory through layers is a path through ℝ^d — a curve in high-dimensional space that starts at the token embedding and ends at the final residual state, which the unembedding maps to the pre-softmax logits. This is, literally, a population trajectory.

Recording from the Residual Stream

Figure 1. The residual stream x_l^(t) ∈ ℝᵈ exists at every layer l and token position t. Layers are numbered from 0 to L − 1 (L = number of layers, e.g. 0–11 in GPT-2 small); x_l enters layer l, and x_L leaves the last one. (Our scripts and figures name each point by the layer whose output it is, after the embedding "emb": x_l here is their point l − 1, and x_L their last layer.) Recording from it produces a dataset of vectors (one per layer, position, and input), directly analogous to a population recording in neuroscience — but with full coverage, zero noise, and perfect repeatability.

What's different from biological neuroscience

Advantages over biology

Full observability. Every dimension at every layer, every token. No sampling bias, no dropped spikes.
Zero noise. The same input gives identical activations. No trial-to-trial variability to average over.
Ground truth connectivity. We have the weight matrices. We know exactly how information flows — in principle.
Arbitrary intervention. Clamp, ablate, patch, or perturb any activation at any layer, any token, any dimension. Optogenetics with infinite precision.
Instant replay. Re-run the same prompt as many times as you want, with or without changes.

Challenges that remain

No clear "neuron" unit. Individual residual stream dimensions aren't interpretable on their own. Polysemanticity means features are distributed across dimensions.
No natural timescale. Layers aren't milliseconds. There's no oscillation, no theta rhythm, no obvious timescale to anchor the dynamics.
Training data is opaque. The model's "experience" (the training corpus) is vast, heterogeneous, and not under our control.
No obvious homolog. Which part of the LLM corresponds to hippocampus? Prefrontal cortex? It's unclear, and forcing the mapping may mislead.
Functional vs. mechanistic understanding. Knowing what a component computes doesn't tell us why it was necessary or how it generalizes.

Open questions

The analogy is productive but generates as many questions as it answers.

Key open questions

What is the right vocabulary? Neuroscience settled on tuning curves, population codes, attractors, and trajectories. Which of these transfer cleanly to LLMs? Which need modification? Which are actively misleading?

What counts as a "finding"? Discovering that V1 neurons respond to oriented edges is a finding because it constrains computational theories. What is the LLM equivalent? Does showing that mid-layer MLPs store factual associations constrain anything important?

How do we validate? Neuroscience validates by intervention and by behavior. LLMs allow arbitrary intervention — but what is the right behavioral read-out? Token probability? Task performance? Out-of-distribution generalization?

Do LLMs have "states"? Neural circuits pass through distinct dynamical states (rest, preparation, execution, recovery). Do LLMs have analogs? Do different prompt types produce qualitatively different trajectory shapes?

What's the right null model? When we see something interesting in neural data, we compare to a null (shuffled spikes, random connectivity). What's the null for an LLM's residual stream? A random transformer of the same size? A model trained on noise?

Learning across trials: A central interest in systems neuroscience is how population activity reorganizes across trials during learning. An analogous question exists for LLMs: how do residual stream dynamics change across training steps? This is the focus of developmental interpretability — a nascent subfield that has already found striking results, including sudden phase transitions in circuit formation (e.g., the abrupt emergence of induction heads) and late-stage generalization long after training loss converges (grokking). It remains underdeveloped relative to post-training analysis, largely because it requires model checkpoints saved throughout training. Few model families release them, but some do: Pythia (154 checkpoints per model) and OLMo.

Neuroscience parallels in Lessons 1–7

Lessons 1–7 compared language models with brains in many places, usually in a box at the end of a tab. This tab collects them in one table, grouped by topic, with the lesson and tab where each first appeared (later tabs that return to it are listed below the link). The last column gives the caveats each lesson stated itself, then further limitations that the lessons did not state.

Three patterns run through the table. Most parallels are structural (similar shapes of computation or representation), not claims of shared mechanism. Many break on the same few differences: brains are recurrent and learn continually with local rules, while a transformer runs one feedforward pass with frozen weights trained by backpropagation. And the strongest parallels are methodological: the same analysis tools work on both systems, with the same pitfalls.

In the model	In the brain	First discussed	Caveats and limitations
Units and learning
1. Artificial neuron	Biological neuron: weights as synaptic strengths, the weighted sum as dendritic summation, ReLU as a firing threshold	L1 · tab 1 The Neuron also L1 · tab 2; L5 · tab 5	In the lessons: No spikes, refractory period or temporal dynamics: a unit is an instantaneous function evaluated once per forward pass (L1 · tab 1); firing rate and membrane potential do not map cleanly onto activations (L5 · tab 5). Also: Dendrites compute nonlinearly (local NMDA and calcium spikes), so one pyramidal cell behaves more like a small network. Real neurons are excitatory or inhibitory (Dale's principle), while an artificial unit mixes signs freely and can flip them in training. GELU units output negative values, which no firing rate can.
2. Activation function	A neuron's input–output (f–I) curve; the sigmoid was modelled on it, ReLU fits a linear-threshold neuron	L1 · tab 2 Activation Functions also L2 · tab 5	In the lessons: Neither captures spiking, adaptation or neuromodulation; they are chosen because they are cheap to differentiate (L1 · tab 2). GELU has no hard threshold, so units are never fully silent (L2 · tab 5). Also: Real f–I curves saturate; ReLU and GELU do not. A neuron's gain changes with neuromodulation and normalization, while the activation function is fixed. GELU and SwiGLU were chosen for optimization, not biology.
3. Gradient descent / backpropagation	Synaptic plasticity with local, Hebbian-style rules	L1 · tab 4 How Networks Learn also L4 · tab 3	In the lessons: "Loose at best": backprop needs a global error signal sent backward through the network, whereas synapses change by local rules (L1 · tab 4); the brain's learning rule is almost certainly different (L4 · tab 3). Also: Exact backprop needs feedback weights that mirror the forward ones (the weight-transport problem) and separate forward and backward phases. Biology seems to use three-factor rules, Hebbian change gated by a neuromodulator such as dopamine. Brains learn continually from far less data (a child hears about 10⁷–10⁸ words; LLMs train on billions to trillions of tokens).
4. Frozen weights of a trained model	The connectome: fixed wiring that supports flexible computation through different activity patterns	L1 · tab 4 How Networks Learn also L5 · tab 5	In the lessons: Biological synapses keep changing (LTP/LTD, neuromodulation, consolidation in sleep), so the connectome drifts; a deployed model's weights are truly static (L1 · tab 4). Also: A connectome is usually a wiring diagram without synaptic strengths, so a model's weights are far more complete than any connectome. Even a complete connectome does not explain behaviour: C. elegans' has been known since 1986 (White et al.), and neuromodulators reconfigure its identically wired circuits.
5. Next-token learning yields rich representations	The brain may arrive at similar representations through a different learning rule	L4 · tab 3 Backprop	In the lessons: Convergence is only "may"; the different learning rule is itself a challenge for using LLMs as neural models (L4 · tab 3). Also: The evidence for convergence (LLM activations predicting fMRI, MEG or ECoG responses to language) is correlational, comes from encoding models limited by noise ceilings and stimulus sets, and partly tracks model size, not only next-word prediction. Similar representations do not imply similar mechanisms.
Coding and representation
6. Distributed representation (many units per concept, each unit in many concepts)	The "grandmother cell" (one cell per concept) as the contrasting local code	L1 · tab 5 Distributed Coding	In the lessons: You cannot read off what a network knows from a single unit (L1 · tab 5). Also: Highly selective concept cells do exist in the human medial temporal lobe (Quian Quiroga et al. 2005), but even they belong to sparse population codes. The 2^d capacity of dense codes ignores noise and readout; real capacity is far lower.
7. Activation patterns and their geometry	Population codes of place, grid and orientation-selective cells; representational geometry	L1 · tab 5 Distributed Coding also L1 · tab 8; L5 · tab 5	In the lessons: LLM activations are noiseless, perfectly reproducible and driven by fully controlled stimuli; in LLMs the sparsity lies in features, not units (L1 · tab 5). Also: Place and grid codes represent continuous, low-dimensional variables (grid cells on a torus); token spaces have no comparable known structure. Biological codes have correlated noise and drift over days. LLM geometry is distorted by a few outlier dimensions and by anisotropy.
8. Token embeddings	The population code of a sensory area; RSA and PCA apply "almost without modification"	L1 · tab 8 Embeddings	In the lessons: None stated. Also: An input embedding is a fixed, context-free table row, not an evoked response; the closer analogue of a sensory response is the contextual residual stream. Sensory responses are noisy and modulated by state and attention.
9. Superposition; features found by sparse autoencoders	Sparse coding in primary visual cortex: a sparse code learned from natural images gives V1-like receptive fields (Olshausen & Field 1996); the overcomplete version of the model is Olshausen & Field 1997	L1 · tab 5 Distributed Coding also L2 · tab 5; L5 · tab 3	In the lessons: The model's units respond densely; the sparsity is in features, not units (L1 · tab 5, L2 · tab 5). The regimes are reversed: V1 has more neurons than inputs, so features can have neurons of their own, while superposition packs more features than neurons, so features share them; what the two share is sparsity (L1 · tab 5, L5 · tab 3). Also: Olshausen & Field is a normative model of receptive fields, not evidence of superposition. Brain sparsity is also driven by energy cost, which LLMs lack. A sparse autoencoder finds a sparse dictionary for almost any data, and its features depend on dictionary size.
10. Residual stream: many overlapping variables in the same dimensions	Mixed selectivity: a prefrontal neuron encoding task context, reward and stimulus at once	L2 · tab 2 Residual Stream	In the lessons: None stated. Also: The residual stream has no privileged basis (no elementwise nonlinearity acts on it), so its dimensions are not neuron-like; an MLP hidden unit is the closer analogue. Mixed selectivity in prefrontal cortex is nonlinear, conjunctive tuning (Rigotti et al. 2013), which linear superposition does not reproduce, and it is usually seen as useful for computation rather than as a side effect of compression.
11. Next-token prediction	Predictive coding: the brain continually predicts its next input	L1 · tab 6 Language as Probability	In the lessons: None stated. Also: Predictive coding is a circuit hypothesis: top-down connections carry predictions and dedicated units carry prediction errors. A transformer has no top-down feedback or error units within a forward pass; its prediction is a training objective, not an inference-time error-correction loop. The evidence for predictive coding in cortex is itself debated.
12. Tokenization: the unit the model sees	Choosing single-unit, multi-unit or population activity as the unit of analysis	L1 · tab 7 Tokenization	In the lessons: None stated (a passing comparison). Also: An experimenter's choice of unit does not change the brain, but the tokenizer is part of the model and fixes what it can perceive (its spelling and arithmetic failures follow from it). Brains receive continuous sensory streams, not pre-cut tokens.
13. Softmax with temperature τ	The Boltzmann / Luce choice rule of decision neuroscience and reinforcement learning (τ = 1/β; low τ exploits, high τ explores)	L1 · tab 9 Next-Token Prediction	In the lessons: None stated. Also: In decision neuroscience softmax is a descriptive model fitted to choices, not a known neural mechanism. Animal exploration is partly directed by uncertainty, not purely random. An LLM's τ is set from outside at decoding time, not adapted by the system.
Architecture
14. The stack of transformer blocks	The hierarchy of sensory cortex (V1 → V2 → V4 → IT; A1 → belt → parabelt)	L2 · tab 1 Overview also L2 · tab 2	In the lessons: Transformer layers are identical in structure; the hierarchy emerges from training alone, not from anatomical specialization (L2 · tab 1). Also: Cortex has dense feedback and lateral connections, so its hierarchy is not strictly serial. Receptive fields grow along the ventral stream, while attention sees the whole context from layer 0. For language, intermediate LLM layers best predict activity in the brain's language network (Schrimpf et al. 2021).
15. Residual updates across layers	Neural population trajectories in a dynamical system; depth plays the role of time	L2 · tab 2 Residual Stream also L5 · tab 5; L8 · tab 3	In the lessons: The closer analogue is the spatial unfolding along the ventral visual stream, not temporal dynamics: a transformer has no recurrence (L2 · tab 2). Whether fixed points, attractors and limit cycles apply is an open question (L5 · tab 5). Also: Each layer has its own weights, so the model is not an autonomous system iterating one rule, and attractors are only loosely defined. There are only 12 steps (GPT-2 small) of uncontrolled size. An LLM has two axes, depth and token position; a brain has one physical time with recurrent dynamics.
16. Residual stream as "working memory"	Working memory as a store whose contents get overwritten	L2 · tab 2 Residual Stream	In the lessons: Presented as a misconception: contributions accumulate and are not overwritten (L2 · tab 2). Also: If anything in an LLM plays the part of working memory, it is the context (KV cache) across tokens, not the stream within one forward pass. Biological working memory is activity sustained over seconds.
17. Attention pattern (recomputed for every input) vs. fixed W_Q, W_K, W_V	Effective connectivity that changes from trial to trial vs. fixed anatomical tracts	L2 · tab 3 Attention	In the lessons: None stated. Also: Effective connectivity is a statistical estimate between brain areas (dynamic causal modelling, Granger causality); attention routes between token positions, not processing areas. Attention weights are not reliable measures of importance: the value vectors and W_O matter too.
18. Transformer attention	Top-down attention gating signals between cortical areas	L2 · tab 3 Attention	In the lessons: Only "reminiscent" (L2 · tab 3). Also: Self-attention has no goal signal from a control area and no capacity limit; it is closer to content-addressable retrieval. Cognitive attention works largely through gain changes and normalization and is capacity-limited. The shared name is mostly historical.
19. Multiple attention heads	Parallel processing streams: ventral "what" and dorsal "where"; auditory pathways	L2 · tab 4 Multi-Head Attention	In the lessons: Cortical streams are physically separate; heads are mathematical abstractions on the same hardware (L2 · tab 4). Also: The two-streams model is itself a simplification with much cross-talk. Heads are often redundant (many can be pruned) and polysemantic, and they sit within one layer, while cortical streams span many areas.
20. Feedforward (MLP) layer, d → 4d → d; attention between positions	The local circuit of a cortical column; long-range projections between areas	L2 · tab 5 Feedforward Layers	In the lessons: GELU is not a hard threshold, so hidden activity is graded and dense (L2 · tab 5). Also: The column as a computational unit is contested, and the laminar circuit is heavily recurrent. Clearer biological expansion-recoding motifs are cerebellar granule cells and the insect mushroom body. The MLP applies identical weights at every position, which has no biological counterpart.
Methods
21. Recording every activation of the model	Electrophysiology: a perfect multi-electrode recording during a computation	L1 · tab 10 Why This Works also L5 · tab 5; L8 · tab 1	In the lessons: Only advantages are listed: full coverage, no noise, perfect repeatability (L1 · tab 5, L5 · tab 5). Also: Full access does not guarantee understanding: standard neuroscience analyses gave misleading answers on a fully known microprocessor (Jonas & Kording 2017). Residual-stream dimensions have no privileged basis, unlike recorded neurons, and the stream is a sum of earlier writes rather than one population's firing.
22. PCA, RSA and dimensionality analysis of activations	The same tools in systems neuroscience; RSA compares models, brain areas and species by geometry	L1 · tab 5 Distributed Coding also L1 · tab 8; L5 · tabs 2, 4, 5	In the lessons: The stimuli are chosen by hypothesis, so contrasts not sampled are invisible; for LLMs one token position (or an average) must be chosen (L5 · tab 4). Also: RSA is a weak constraint: different mechanisms can give similar dissimilarity matrices, and scores depend on the distance measure and noise ceilings. PCA on LLM activations is dominated by outlier dimensions unless normalized. UMAP and t-SNE distort global distances.
23. Activation patching	Optogenetics and microstimulation: causal tests, not correlations	L5 · tab 5 Neuro Parallels	In the lessons: None stated. Also: Patching swaps in the exact activation from another input; optogenetics drives genetically defined cells coarsely, often outside their normal range. In both systems other components can compensate (redundancy in brains, "self-repair" in LLMs), and a causal effect does not pin down the mechanism.
24. Rank-one weight edit (ROME); MLP as associative memory	Forming a new associative memory trace (a Hebbian-like, outer-product change across many synapses)	L7 · tab 7 Model Editing	In the lessons: Facts are distributed across layers; other keys are kept only approximately; sequential edits degrade the model; editing success is weaker evidence for localization than it looks (Hase et al. 2023); a rank-one update changes every entry of W_down, not one synapse (L7 · tab 7). Also: Biological memories form through plasticity and consolidation, not one closed-form step. The damage from sequential edits resembles catastrophic interference, which brains reduce with complementary learning systems.
25. Interpretability's open methodological problems	Systems neuroscience's own unfinished move from isolated findings to theory and standards	L5 · tab 5 Neuro Parallels	In the lessons: None stated (a rhetorical comparison). Also: The comparison is about the state of two fields, not a mechanism; Jonas & Kording (2017) is the classic critique that applies to both.
Cognition and behaviour
26. The reversal curse: trained on "A is B", the model does not learn "B is A"	A person who learns A is B automatically knows B is A	L6 · tab 4 Directional Knowledge	In the lessons: None stated for the human side. Also: Human recall is not fully symmetric either: backward recall of sequences is hard, and forward and backward cued recall can differ. People often infer B is A by reasoning, and LLMs can too when both facts are in the context; the curse concerns knowledge stored in the weights.

Parallels the lessons imply but do not draw

Lesson 7 in particular uses methods and terms with obvious neuroscience counterparts without naming them. For completeness:

In the model	In the brain	Where	Main limitation
Ablation	Lesions and inactivation (muscimol, cooling)	L7 · tabs 1, 5	Ablation is exact and instant, with no recovery or reorganization; it can push activations outside anything seen in training.
Linear probe	Decoding stimulus or choice from population recordings	L7 · tab 2	Decodable does not mean used, in both systems (L7 · tab 2 notes this for LLMs).
Steering vectors	Microstimulation biasing perception (Salzman, Britten & Newsome 1990)	L7 · tab 6	Microstimulation drives an anatomical cluster; steering adds a direction across the whole population, exactly and reversibly.
Polysemantic neurons	Mixed selectivity	L5 · tab 2; L7 · tab 1	Mixed selectivity is usually seen as useful for computation, polysemanticity as a side effect of superposition.
A single SAE feature (the Golden Gate Bridge)	Concept cells in the medial temporal lobe (Quian Quiroga et al. 2005)	L7 · tab 8	A feature is a direction, not a unit; both are characterised from top-activating examples.
Circuits; "inhibition" heads; induction heads appearing abruptly	Neural circuits; inhibitory interneurons; developmental stages	L7 · tab 5	An "inhibition" head writes a negative contribution, it is not an inhibitory cell type; a training phase transition is not a critical period.
Lost in the middle (U-shaped recall over the context)	The serial-position curve of human free recall	L6 · tab 1	Human primacy and recency come from rehearsal and working memory; the LLM's comes from causal attention and position encoding.
"Hallucination", "confabulation"	Perception without a stimulus; confabulation after memory-system damage	L6 · tabs 2, 3	LLM errors are false statements, not false perceptions, and there is no separate truth-checking system that has broken.

Lesson 3 (Forward Pass) draws no neuroscience parallels.

Transformer Dynamics: A neuroscientific approach to interpretability of large language models

Fernando & Guitchounts · arXiv:2502.12131, 2025 · LLM residual stream as dynamical system

Background: the population dynamics framework

This paper applies a framework developed in systems neuroscience — the population dynamics view — to LLMs. The framework emerged from simultaneous recordings of many neurons in motor cortex and was synthesized in a landmark review (Vyas, Golub, Sussillo & Shenoy, Annual Review of Neuroscience, 2020). Rather than asking "what does this neuron encode?", it asks: "what shape does the population activity trace through high-dimensional space, and what does that shape reveal about the computation?"

When you record from N neurons simultaneously, you get a point in ℝ^N at each moment. The sequence of points over time is a trajectory. In practice, these trajectories tend to live on a neural manifold — a low-dimensional subspace capturing the computationally relevant variation. Motor cortex during reaching, for instance, uses only a handful of dimensions out of hundreds of recorded neurons.

Two concepts from this tradition are especially relevant here. First, attractors: regions of state space that the trajectory flows toward and stays near, arising from the learned connectivity of the network rather than any explicit programming. Second, rotational dynamics (Churchland et al. 2012): in motor cortex, population trajectories trace near-circular orbits when projected onto their top principal components, generating the sequential muscle activation needed for smooth reaching. Fernando & Guitchounts find rotational structure in Llama 3.1 8B as well — but in the phase portraits of individual units rather than in population principal components — and whether this reflects the same functional role remains an open question, since the transformer is feedforward rather than recurrent. In a transformer, "time" maps to layer depth, not to milliseconds; the geometric tools carry over, but their functional interpretation may not.

Overview

This paper directly imports the neuroscience dynamical systems framework into LLM analysis. The authors treat the residual stream of Llama 3.1 8B as a high-dimensional dynamical system and ask: what is the geometry of token trajectories across layers? The results are strikingly parallel to what has been found in motor cortex — cross-layer continuity, rotational structure, acceleration and deceleration phases, and a mid-network pseudo-attractor.

Methodology

Model

The primary model is Llama 3.1 8B (L = 32 transformer layers, hidden dimension D = 4,096). Gemma 2 was used in a preliminary analysis but results are not reported. The residual stream is extracted at two checkpoints per layer: before the attention sublayer's layer norm (h_l^Attn) and before the MLP sublayer's layer norm (h_l^MLP). This yields 2L = 64 interleaved sublayer states per sequence.

Tensor shape: B × 64 × 4096 where B = batch size, 64 = sublayer checkpoints, 4096 = residual stream dimension Each of the 4096 dimensions is referred to as a "unit."

Input data

Text is drawn from WikiText-2-raw-v1, a standard benchmark corpus of Wikipedia articles. Sequences are filtered to lengths between 100 and 500 characters. Each sequence is prepended with a beginning-of-sequence (BOS) token. All analyses are performed on the last token position only — the "readout" position that determines the next-token prediction. Most analyses use N = 1,000 sequences (the paper calls them both "samples" and "data batches"); the autoencoder (analysis 7) uses a separate, much larger set. A shuffled control condition randomly permutes all tokens after the BOS token, breaking syntactic and semantic structure while preserving token identity. A perturbation experiment uses the prompt "I'm sorry, Dave. I'm afraid I can't do that."

Analyses

Per-unit Pearson correlation. For each of the 4,096 units, the Pearson correlation of its activations across all samples is computed for every pair of sublayer checkpoints (l, m). This produces a distribution of correlations spanning all layer-pair combinations, showing how long individual unit activations remain stable as the stream evolves.

Cosine similarity between consecutive sublayers. CS_l = (h_l · h_l+1) / (‖h_l‖ ‖h_l+1‖). Measures the angle between adjacent residual stream vectors — a value near 1 means the state barely changed direction; a lower value means a sharper turn in high-dimensional space.

Velocity. ‖V_l‖ = ‖h_l+1 − h_l‖₂ — the Euclidean distance between adjacent sublayer states. Where cosine similarity captures the direction of change, velocity captures its magnitude.

Phase portraits and rotation counting. For each unit i, the authors plot its activation a_lⁱ (x-axis) against its layer-wise gradient (y-axis), forming a phase portrait. Trajectories are centered by subtracting the initial value. Tangent angles are computed with arctan2 and adjusted for ±π wrapping. The total rotation count per unit is R = (1/2π) Σ Δθ_l. A null distribution is generated by shuffling layer order 1,000 times per unit.

Mutual information (MI) between consecutive sublayers. MI is estimated per unit using Gaussian kernel density estimation, in nats. Formula: MI = Σ p(x,y) log( p(x,y) / (p(x) p(y)) ), with a small constant (10⁻¹⁰) added for numerical stability. Computed separately at attention → MLP and MLP → attention transitions to isolate the information-theoretic signature of each sublayer type.

PCA via SVD. The residual stream tensor (N × 2L × D) is reshaped to a (2NL) × D matrix and mean-centered. The top right singular vectors are the principal axes. Explained variance per component: r_k = σ_k² / Σσ_i². Per-layer explained variance is also computed using the top 100 components.

Compressing autoencoder (CAE). To visualize trajectory geometry in 2D, a bottleneck autoencoder is trained: input dimension 4,096, bottleneck dimension 2, with hidden layers following a geometric width progression. Training uses Adam (lr = 10⁻³), MSE loss, up to 100 epochs with early stopping. The autoencoder was trained on 85k batches and tested on 15k batches, each of 64 samples.

Perturbation ("teleportation"). A 10 × 10 grid of points is spread across PC space and mapped back to activation space as x = z V₂^⊤ + μ (z: a grid point’s two PC coordinates; V₂: the top two principal axes as columns; μ: the mean). These reconstructed activations replace the residual stream at chosen layer depths (layers 0, 7, 15, 23, 31), and recovery is measured as MSE against an unperturbed control over the following 12 sublayers. This tests whether the system has attractor-like dynamics — whether it pulls perturbed states back toward a canonical trajectory.

Finding 1: High cross-layer continuity (r > 0.8, rising with depth)

For each of the 4,096 units, its activations at adjacent sublayers are highly correlated across inputs: r > 0.8 even in the earliest layers, and rising with depth. The residual stream doesn't change drastically from layer to layer — each layer makes a relatively small modification to the existing state, and attention sublayers change it less than MLP-to-next-layer transitions. This is similar to the smooth population trajectories seen in motor cortex — abrupt discontinuities would suggest a phase transition or qualitative change in computation.

Finding 2: Velocity profile and attractor-like dynamics

Two related findings point to structurally distinct phases across the network.

Velocity profile. Velocity (‖h_l+1 − h_l‖₂) is roughly constant across early and mid layers, then rises, most steeply in the last third of the model. Note that cosine similarity — a separate measure — increases as a function of depth, meaning adjacent layers become more directionally aligned as you go deeper. The two measures are compatible: later layers can change the residual stream by a larger magnitude (higher L₂ distance) while shifting its direction less abruptly (higher cosine similarity).

Attractor-like dynamics. The authors test whether the network has attractor-like dynamics by "teleporting" activations — replacing the residual stream at a chosen layer with points drawn from a grid in PCA space, then measuring how closely trajectories return to the unperturbed mean over the following 12 sublayers. Perturbations at the extremes fail: layer 0 produces highly variable dynamics; layer 31 causes "even more drastic" deviation. (The paper calls layer 31 "penultimate", but with 32 layers numbered from 0, as layer 0 implies, it is the last.) Mid-layer perturbations at layers 7, 15, and 23 show the most robust recovery. The abstract and section heading place the attractor-like dynamics in "lower layers"; the recovery results themselves point to layers 7–23. The Discussion characterizes this as a "pseudo-attractor" — the system tends to pull perturbed states back toward a canonical trajectory, but only when the perturbation occurs before the final output phase.

Finding 3: Rotational dynamics in single units (~10.74 rotations per unit)

This finding is about individual units, not the whole residual-stream vector. For each of the 4,096 residual-stream units, the authors draw a phase portrait: the unit's activation (x-axis, centered on its initial value) against its layer-to-layer change, its "gradient" over the 64 sublayer states (y-axis). They then count how many times the direction of travel along that 2D trajectory turns through a full circle (analysis 4 above).

On average, units complete about 10.74 rotations across the network (the mean over units, given in the paper's Fig. 2B caption). When the layer order is shuffled (1,000 shuffles per unit), the count drops to approximately zero, so the rotations depend on the order of the layers rather than arising by chance. The paper reports no spread (SD or range) and no test statistic for this comparison.

It is not the total angle that the whole residual-stream vector turns through. The paper never sums angles between consecutive whole-vector states; for the whole vector it reports cosine similarity and velocity instead (Finding 2).

Because residual-stream units are not a privileged basis, a unit's rotations need not reflect any rotation of the population state, unlike motor cortex, where rotations appear in the population's principal components; and the count of 10.74 depends on the 64 sublayer steps and on how the layer-to-layer "gradient" is estimated.

Phase portrait of unit i: ( a_l − a₀ , da/dl ) for the l = 0 … 63 sublayer states Tangent angle: θ_l = arctan2( Δy_l , Δx_l ) (adjusted for ±π jumps) Rotations: R_i = (1/2π) Σ_l Δθ_l mean over the 4,096 units ≈ 10.74 (shuffled layer order ≈ 0)

Summary of findings

Finding	What was measured	Result
Cross-layer continuity	Pearson r of each unit's activation between adjacent sublayers, across inputs	r > 0.8 from the earliest layers, increasing with depth
Velocity profile	L₂ distance ‖h_l+1 − h_l‖₂ per layer step	Roughly constant early–mid; steepest acceleration in last third
Attractor-like dynamics	Teleportation recovery (MSE vs. unperturbed control)	Best recovery at mid-layer perturbations (layers 7, 15, 23); extremes fail
Rotational dynamics	Rotations of each unit's phase portrait (activation vs its layer-to-layer change, 64 sublayers)	~10.74 rotations per unit on average in Llama 3.1 8B; ≈ 0 with shuffled layer order

Open questions & future work

The paper's analyses all use a single input distribution — Wikipedia prose in English — and a single model (Llama 3.1 8B). This leaves several natural next steps open.

Stimulus variation. Do the dynamics look the same across different kinds of input? The paper does not compare RS trajectories for, say, English vs. French text, or formal vs. informal register; it defines a shuffled-token control but reports no results for it. The authors acknowledge this gap, noting that the low dimensionality they observe may partly reflect "the relative simplicity" of encoding high-probability Wikipedia sequences. Tasks requiring in-context learning or multi-step reasoning might reveal richer or qualitatively different trajectories.

Whole-sequence dynamics. All main analyses operate on the last-token position only. How representations evolve across the token dimension — not just the layer dimension — remains unexplored here.

Cross-model comparisons. A single model can't tell you whether the 10.74-rotation figure, the velocity acceleration pattern, or the attractor depth are universal or idiosyncratic. Comparing models of different scales, depths, and training objectives is needed to know whether these are architectural constants or incidental properties of one checkpoint.