How large language models work, and how to study them
Nine lessons, from the artificial neuron to hands-on interpretability experiments on GPT-2. Lessons 1–8 explain the ideas with interactive figures; Lesson 9 applies the methods to a real model, with MATLAB scripts that reproduce every result.
The code (Matlab code, Python code) and the instructions for running it are in the repository github.com/sotmasman/LLM-mechanics ; see its README.
Lesson 1
Neural Networks and Language as Probability
What a neural network computes, how it learns and represents information, and how text becomes tokens, vectors and next-token probabilities.
Lesson 2
The Transformer
Almost every modern language model is built from a stack of transformer blocks (state-space models such as Mamba, and hybrids, are the main exceptions). This lesson traces exactly how information flows through one block — from …
Lesson 3
From Tokens to Probabilities
With every architectural component in place, this lesson traces a single sentence through the complete transformer — from raw text to a next-token probability distribution — and then steps through how a model actually generates …
Lesson 4
Training & Optimization
How a transformer goes from random weights to a language model — loss functions, gradient descent, scaling laws, and what training dynamics reveal about learning itself.
Lesson 5
Representations
What does a transformer actually learn? What computational structures emerge when LLMs learn, and what is the geometry of this feature space?
Lesson 6
What LLMs Get Wrong — and Why
Modern language models are remarkably fluent. Fluency is not accuracy. This lesson catalogues the systematic failure modes baked into the architecture, the training objective, and the fine-tuning process — and explains why each …
Lesson 7
Mechanistic Interpretability
Reverse-engineering the algorithm learned by gradient descent — probing what the network knows, tracing how it computes, and finding the circuits that implement specific behaviors.
Lesson 8
LLMs as Neural Systems to Study: The Electrophysiology Angle
What does it look like to study a language model the way a neuroscientist studies a brain? From the fundamental analogy to the geometry of residual stream dynamics.
Lesson 9
Playing with GPT-2
Hands-on experiments with a real language model — activations, attention, and the residual stream under the microscope.