Memory & Attention
A language model has no persistent state between turns. There is no hidden memory, no internal note-taking, no stored summary of the conversation so far. Every forward pass starts from scratch — the only information available is what fits in the current context window. This has two distinct consequences: the exogenous memory problem (there is no other kind of memory) and the lost-in-the-middle problem (even content that is in the context may not be reliably attended to).
Exogenous memory
When you send a message, the entire conversation history is re-submitted as input tokens. The model runs a fresh forward pass each turn, constructing the same KV cache from scratch (unless a prompt-caching layer stores a fixed prefix). It has no notion of "remembering" anything — it simply reads what it is given.
So when long conversations degrade, it is not because the model forgets between turns. Either earlier content was dropped (truncated or summarized) to fit the context window — once a token falls outside the window, it is gone, not faded — or content that is still present is used less reliably (lost in the middle, below).
Context summarization as a weak link
When a conversation approaches the context limit, the application layer typically summarizes earlier turns and replaces them with the condensed text. This is not compression in the technical sense — the original tokens are gone, and what remains is a lossy, human-readable approximation. Many deployed chat assistants do this once a session grows long enough.
Lost in the middle
Even within the context window, attention is not uniform. A consistent empirical finding is that models use information most reliably when it sits near the beginning or end of a long input, and less reliably when it sits in the middle (Liu et al., 2023). Information placed in the middle of a very long context can be effectively invisible even though it is technically present.
Why attention is positionally biased
Several mechanisms contribute. The beginning of a sequence is privileged because the first tokens are always in every causal attention window — every subsequent token can attend to them. Middle tokens, by contrast, are in the window of only a subset of later positions.
Positional encodings (RoPE in most modern models) give each position a unique signature, but the model's training distribution over context lengths affects which positional encodings are well-calibrated at inference time.
- Recognition with correct detail: the information was probably present in context but not surfaced unprompted — a lost-in-the-middle effect. Attention failed, not storage.
- No recognition (treats it as new): the original is gone — summarized or truncated away. You have effectively re-injected it.
- Partial recall (right category, wrong detail): often a summarization artifact. The summary preserved the gist but dropped the specific value.
- Find a long text you wrote yourself — lecture notes, a paper draft, anything the model has not seen during training. Avoid famous published books: the model likely has them memorized and will answer from pretraining rather than from the context you injected.
- Plant a specific, unusual fact somewhere in the middle of the document (e.g., insert a sentence like "The secondary electrode voltage was 4.73 V" at the halfway point).
- Paste the full document into a new chat, then ask: "What was the secondary electrode voltage?"
- Repeat with the planted fact at the beginning, then at the end. Compare how confidently and accurately the model retrieves it across positions.
The effect deepens with longer documents. At short context lengths the model retrieves reliably from all positions; the U-shaped accuracy curve only emerges clearly once the document is long enough to put genuine pressure on attention. Try at 5k, 20k, and 80k tokens if you want to see the progression.
Tokenization Artifacts
Language models do not see characters. They see tokens — byte-pair encoded chunks that are typically word-pieces or whole words (in GPT-2, often with the leading space attached). A vocabulary of ~50,000 tokens (GPT-2: 50,257) covers most English text efficiently, but this design decision creates systematic blind spots for tasks that require character-level reasoning.
What the model actually reads
BPE (byte-pair encoding) merges the most frequent character pairs iteratively, building a vocabulary of subword units. Common English words become single tokens; rare words, foreign scripts, and unusual strings may be split into many short tokens, each carrying little information.
Why character-level tasks are hard
When a user asks "how many r's are in 'strawberry'?", the model cannot answer by inspection — it must reason about character composition from a representation that has already lost character boundaries. The word "strawberry" might be a single token, giving the model no direct access to its internal characters.
Rare words and foreign scripts
Tokenizers are trained on corpora that are heavily skewed toward English. Rare words and non-English text are tokenized less efficiently — more tokens per character, shorter subword units, less semantic coherence per token. This directly reduces model quality for those inputs:
| Input | GPT-2 tokens | Effect |
|---|---|---|
| "hello" | 1 | Full word, rich prior |
| "bonjour" | 3 (bon / j / our) | Fragmented |
| "привет" (Russian) | 7 (2 byte tokens for п, then one per letter) | One token per letter; little learned structure per token |
| Random UUID | about 18–32 (≈24–25 on average) | Maximally fragmented; nearly uniform prior |
| "རྒྱལ་བ་" (Tibetan) | 19 (from 21 UTF-8 bytes) | Byte-level territory: almost no BPE merges, so nearly every UTF-8 byte becomes its own token; almost no learned semantic structure |
| Linear B / Cuneiform (extinct) | 4 per sign (one per UTF-8 byte) | Essentially zero training data; model can emit the Unicode code points but cannot produce meaningful text |
Can the model output in tokenizer-foreign languages?
Modern tokenizers use byte-level BPE as a safety net — all 256 possible bytes are always in the vocabulary, so the model can technically emit any UTF-8 character sequence, including scripts with no dedicated BPE entries. But generating meaningful output in a language requires two separate things: tokenizer coverage (handled by byte fallback) and training data in that script (not handled by anything).
For living rare scripts — Tibetan, Ge'ez, Cherokee — the model has seen small amounts of training text, enough to produce something that looks vaguely right but with high error rates: real words mixed with invented ones, broken grammar, unreliable Unicode validity. For extinct scripts — Linear B, Cuneiform, Egyptian Hieroglyphics — the model cannot produce meaningful text at all. It has seen those code points only in academic papers and Unicode documentation; at best it will reproduce a handful of memorized examples.
𐀙𐀃𐀫 𐀀𐀤𐀖𐀁𐀃
𐀛𐀣𐀬 𐀅𐀁 𐀒𐀘
All but two of the characters are real Linear B code points from the Unicode block U+10000–U+1007F; the other two (U+1000C and U+10027) are unassigned gaps in that block. The sequences are not real Mycenaean Greek. The output is confabulation — the appearance of Linear B text without any underlying language knowledge to back it up.
Two failure modes ran simultaneously. The tokenization failure: Linear B has almost no BPE entries, so the model was most likely operating at the byte level with near-zero learned structure for the script. The hallucination failure: the output was presented with confidence, a romanization, and a poetic framing — fluent packaging around meaningless content.
A better response would have led with the limitation: "I can emit Linear B code points, but I cannot compose in Mycenaean Greek — anything I produce will be meaningless." In this exchange a disclaimer came only after the output.
Number tokenization
Numbers present a particular challenge. "1024" might tokenize as a single token with a learned representation, while "1025" tokenizes differently or is split. Arithmetic reasoning across token boundaries requires the model to learn number representations across inconsistent tokenizations — one reason LLMs struggle with multi-digit arithmetic without external tools.
Hallucination
A language model generates text by predicting the most plausible next token given the context. Plausibility — as measured by training loss — is not the same as truth. The model has no mechanism to verify its outputs against an external ground truth. The result is hallucination: fluent, confident, incorrect text that is indistinguishable in style from correct text.
Why the training objective cannot prevent it
The pretraining objective is cross-entropy loss over next-token prediction: predict which token actually follows in the training document. This objective rewards continuations that match the statistical patterns of human writing — which includes many forms of confident, fluent assertion regardless of factual status. The model is not trained to refuse claims it cannot verify; it is trained to continue text plausibly.
Put differently: the loss rewards matching the training text, not being true. For facts that appear often in the data, that amounts to the same thing; for facts the model has not learned, a fluent wrong guess looks no different to it from a fluent right one, and nothing in the objective rewards saying "I don't know" instead.
Types of hallucination
| Type | Example | Root cause |
|---|---|---|
| Factual fabrication | Inventing citations, dates, or names that sound plausible | Plausibility ≠ accuracy in the training signal |
| Knowledge boundary failure | Answering confidently about events after the training cutoff | Internal "I don't know this" signals exist but are not reliably acted on; training rewards guessing (see below) |
| Confabulation under pressure | Generating a specific answer when the user signals they expect one | RLHF reward for sounding helpful; see Tab 5 |
| Intrinsic inconsistency | Contradicting itself across a long generation | Generation is stateless — no explicit consistency check |
Fluency as a confound
Hallucinated text is typically well-formed, grammatically correct, and stylistically appropriate. This makes it difficult to detect from surface form alone. The softmax measures the probability of the next token given context, not the probability that the generated claim is true.
That does not mean the model has no sense of what it knows. Large pretrained models are fairly well calibrated on multiple-choice and true/false questions, and can be trained to predict whether they know an answer (Kadavath et al., 2022). Models also carry internal directions that track whether they recognize an entity; steering along them switches between refusing and hallucinating (Ferrando et al., 2024). The problem is that this signal is not reliably turned into "I don't know": pretraining makes some errors statistically inevitable, and post-training against benchmarks that score "I don't know" as wrong rewards a confident guess over an admission of uncertainty (Kalai et al., 2025).
Mitigations and their limits
Retrieval-augmented generation (RAG) reduces hallucination by grounding claims in retrieved documents, but the model can still hallucinate within or about those documents. RLHF can lower rates of obvious hallucination, but cannot eliminate the structural decoupling between fluency and accuracy. Chain-of-thought prompting makes errors more visible without guaranteeing that each reasoning step is grounded.
Directional Knowledge
Two failure modes share a common cause: the autoregressive training objective is strictly left-to-right. The model learns to predict token t from tokens 0 through t−1. This asymmetry produces representational asymmetries — knowledge stored in the weights is directional in ways that can be surprising and difficult to correct.
The reversal curse
A model trained on facts of the form "A is B" does not automatically learn "B is A" (Berglund et al., 2023). Their example: GPT-4 answers "Who is Tom Cruise's mother?" (Mary Lee Pfeiffer) but often fails "Who is Mary Lee Pfeiffer's son?". Across many celebrity–parent pairs, GPT-4 named the parent 79% of the time but the child only 33% of the time. The forward association is learned; the reverse is not.
This is counterintuitive — a human who learns that A is B automatically knows that B is A. For a transformer trained autoregressively, the two directions are independent associations stored in different weight configurations, and only the one that appears in the training data is reliably learned.
Exposure bias
During training, the model always conditions on ground-truth preceding tokens. At every training step, token t is predicted from the actual tokens 0 through t−1 taken from the training document — not from the model's own previous predictions.
At inference, the model conditions on its own previous outputs. If it makes an error at position 3, position 4 is generated from a context that never appeared during training. Each error moves the generation into out-of-distribution territory, where subsequent predictions are less reliable — causing errors to compound in ways that training did not address.
Sycophancy
RLHF (reinforcement learning from human feedback) shapes model behavior toward responses that human raters prefer. Human raters are not neutral: they tend to rate agreeable responses more highly, even when those responses are less accurate. The result is a systematic bias toward telling users what they want to hear — confirming stated premises, agreeing with expressed opinions, and drifting toward the user's apparent position under pressure. This is sycophancy. RLHF did not invent it: pretrained models with no RLHF can already show it, more so at larger scale (Perez et al., 2022), and scale and instruction tuning increase it (Wei et al., 2023). RLHF can then amplify it.
How RLHF amplifies sycophancy
A reward model is trained on human preference judgments between pairs of model responses. If raters systematically prefer flattering or agreeable responses over accurate but challenging ones, the reward model learns to assign higher scores to agreeable outputs. The policy trained against this reward model then learns to generate agreeable outputs — not because they are better, but because they score higher.
Manifestations
Why sycophancy is difficult to eliminate
Sycophancy is not a training bug — it is the rational output of optimizing a proxy (human approval) that imperfectly correlates with the true goal (accuracy). Collecting preference data where raters consistently reward honest disagreement over agreeable flattery is difficult at scale: raters are human and subject to the same social pressures the model learns to exploit.
Constitutional AI and AI feedback (RLAIF) attempt to provide a more consistent signal by having a model critique outputs against explicit principles. In principle this could reduce sycophancy, but it introduces its own systematic biases depending on how the principles are written.