This course asks how a collection of numerical calculations can turn a question into a useful answer. We will build the explanation from probability, vectors, matrices, and calculus. The models in the worked examples are deliberately tiny; real language models use far larger vocabularies, vectors, and networks.

1.1 What is a language model?

A language model assigns probabilities to sequences of tokens. A token is a unit of text: it might be a word, part of a word, punctuation, or a representation of bytes. We will first use whole words to keep the arithmetic visible.

Consider the unfinished sentence:

The pianist played the ...

Many continuations are possible. A toy model could assign these probabilities:

The values are nonnegative and sum to 1. They describe a distribution over possible next tokens, conditional on the preceding text. They do not measure whether a whole factual statement is true.

The vertical line means “given”. The subscript $t$ identifies a position in the sequence. Changing the preceding text can change the distribution. After “The drummer played the”, a useful model would generally give different scores.

Interactive lab

A distribution changes with context

Choose a promptThe pianist played the …The drummer played the …

These are invented word-level distributions. They illustrate conditional prediction, not a live language model.

1.2 From one token to a whole sentence

The probability chain rule lets us decompose a sequence probability into conditional probabilities:

For a fixed prompt $C$, suppose our toy model gives:

Then the probability of that three-token continuation is:

These are conditional probabilities, not an assumption that the tokens are independent. The next distribution is recalculated after each selected token. A completed sequence also needs an end condition; here we are calculating only the probability of these three specified tokens.

Interactive lab

Multiply conditional probabilities

P(piano | prompt)

0.6

P(beautifully | … piano)

0.5

P(. | … piano beautifully)

0.8

Sequence probability0.240

Try halving one factor while holding the others fixed. The product also halves.

1.3 A model is a parameterised function

We write a model's prediction as $P_\theta(w_t\mid w_{<t})$. The symbol $\theta$ stands for all its adjustable numerical parameters, and $w_{<t}$ means the preceding tokens.

A conventional language model might count how often words occur together. A neural language model instead calculates predictions through a learned numerical network. Its parameters determine how input representations are transformed into output scores.

“Large” usually refers to the substantial scale of these networks and their training. It does not change the underlying requirements: represent the input, calculate scores, convert them to a distribution, and choose a continuation.

1.4 Training and generation are different processes

During training, the model sees examples with known next tokens. A loss function measures how much probability it assigns to the actual continuation. An optimiser changes the parameters to reduce average loss. Lesson 5 develops the calculus behind this process.

During ordinary generation, the trained parameters are held fixed. The model calculates a distribution, selects a token, appends it to the context, and repeats. Sampling can produce different outputs even when the parameters are unchanged. Lesson 6 explains the selection rules.

Learning from an example in the current conversation is often a change in behaviour caused by context. It does not, by itself, mean the deployed model has updated its weights. Applications may separately save information or run later training processes.

1.5 Why prediction can be useful

Predicting a continuation of a proof, a translation, or a program can benefit from learning relationships within those domains. The training objective concerns tokens, while the computations that improve prediction can be much richer than counting adjacent words.

This does not guarantee that every answer is correct. A fluent continuation can contain an error or an invented fact. Keep three questions separate throughout the course: what calculation was performed, what behaviour it supports, and what evidence establishes that the answer is true.

Check your understanding

Lesson 1 checkpoint

The next three conditional probabilities are 0.6, 0.5, and 0.8. What is the probability of that continuation?

A. 1.90B. 0.24C. 0.63D. 0.80

Check answer

The central insight from Lesson 1

An autoregressive language model builds a sequence through conditional next-token predictions. Its learned parameters and its current context jointly determine those predictions. Our next step is to see how text becomes numerical input.

The network cannot multiply words directly. It needs numerical representations. This lesson follows text through tokenisation, token IDs, embedding lookup, and vector comparisons.

2.1 Splitting text into tokens

A tokenizer uses a vocabulary and encoding rules to map text into a sequence of token IDs. Common tokenizers work with subwords or bytes, so a familiar word can be one token while a rarer word is split into several. Spaces, punctuation, and letter case can affect the result.

For illustration, “The pianist played the piano.” might become:

The · pianist · played · the · piano · .

The spaces shown inside these example pieces are intentional. This is an invented segmentation, not the output of a particular production tokenizer. Different tokenizers can segment the same text differently.

Interactive lab

Inspect a toy tokenisation

Text to encode (up to 200 characters)Encoding demonstrationGreedy matching against a tiny vocabularyOne Unicode code point per piece

The toy vocabulary includes “The”, “ pianist”, “ played”, “ the”, and “ piano”. Unmatched characters fall back to individual code points. This is not BPE or the tokenizer of any particular LLM.

2.2 Token IDs are labels

Suppose a toy vocabulary assigns piano ID 0, violin ID 1, and guitar ID 2. These integers identify rows in a table. ID 2 does not mean that guitar has twice as much musical meaning as violin, and nearby IDs need not have related meanings.

The table is the embedding matrix. Each row contains a learned vector:

The lookup for piano returns:

We have displayed a row as a column vector for convenience. Both conventions are valid when matrix dimensions are kept consistent. A one-hot row vector $\mathbf{u}=[1,0,0]$ performs the same lookup by matrix multiplication: $\mathbf{u}E=[0.2,0.7,-0.1]$.

Interactive lab

Look up an embedding

Tokenpiano · ID 0violin · ID 1guitar · ID 2

The three vectors are the same invented embedding table used again in Lesson 5.

2.3 Dot products compare vectors

For vectors of the same dimension, the dot product multiplies corresponding components and adds them:

For our piano and violin embeddings:

A dot product depends on both the directions and lengths of the vectors. The Euclidean length, or norm, is:

For piano the length is $\sqrt{0.54}\approx0.735$, and for violin it is $\sqrt{0.49}=0.700$.

2.4 Cosine similarity compares directions

Divide the dot product by the product of the lengths:

Our result is $0.50/(\sqrt{0.54}\times0.70)\approx0.972$. The invented piano and violin vectors point in similar directions. For piano and guitar, the dot product is $0.19$, and cosine similarity is approximately $0.287$.

Cosine similarity ranges from −1 to 1 for nonzero real vectors. It is undefined for a zero vector because its denominator is zero. Similarity can be useful for retrieval, but it is not a universal measure of meaning or truth.

Interactive lab

Explore vector direction and length

Second vector: x

1

Second vector: y

1

Dot product with [1, 0]1.00

Cosine similarity0.707

The fixed vector is [1, 0]. The plot uses only two dimensions so direction is visible. At [0, 0], cosine similarity is undefined.

2.5 What do the dimensions mean?

In our table, the numbers are invented for teaching. In a real model they are learned. Individual dimensions do not generally come with labels such as “musicality” or “size”. Useful relationships can be distributed across many coordinates.

Changing the coordinate system can preserve many geometric relationships while changing the interpretation of each coordinate. We should be cautious about reading a simple human concept into one number.

2.6 Context changes a token's representation

The initial embedding for a token is usually obtained from a fixed lookup table. Later Transformer layers produce representations that depend on the surrounding context.

Compare “She sat on the river bank” with “She deposited money at the bank”. The initial token may be the same, while the contextual representations differ. Attention, in Lesson 4, is a major mechanism for combining information from different token positions.

Position matters too. “The dog chased the cat” and “The cat chased the dog” contain similar tokens but describe different events. The model therefore needs information about order in addition to token identity.

2.7 Embeddings are trainable parameters

An embedding matrix with vocabulary size $V$ and vector dimension $d$ has $Vd$ entries. A toy vocabulary of 1,000 tokens with 64-dimensional embeddings needs 64,000 numbers. Real models can use much larger values.

These numbers are adjusted during training along with the network's other weights. Lesson 5 returns to our piano vector and works through a numerical update.

Check your understanding

Lesson 2 checkpoint

What does a token ID tell us?

A. The numerical amount of meaning in a wordB. Which entry to look up in a vocabulary or embedding tableC. The probability that the token is trueD. Its distance from all other words

Check answer

The central insight from Lesson 2

Tokens identify pieces of text; embeddings turn those identifiers into learned vectors. Geometry gives us useful calculations, and the network then transforms the vectors into representations that depend on context.

We now have vectors representing tokens. A neural network transforms these vectors through learned weights, biases, and nonlinear functions. Small examples let us follow every operation.

3.1 A neuron is a weighted calculation

A simple neuron first calculates a weighted sum and adds a bias:

For $x_1=2$, $x_2=1$, $w_1=0.5$, $w_2=-1$, and $b=0.1$:

The weights determine how each input contributes. The bias shifts the result even when the inputs are zero. These are numerical parameters, learned from data rather than individually written as rules.

3.2 Several neurons form a matrix operation

Collect the weights into a matrix:

Then:

A matrix with $m$ rows and $d$ columns maps a $d$-component column vector into an $m$-component column vector. Here a 2 × 2 matrix maps two input components to two output components.

The numerical work consists of multiplication and addition. Larger matrix operations are particularly suitable for parallel hardware.

3.3 Why nonlinear functions matter

If we only stack affine transformations, they collapse into one affine transformation:

Adding more such layers would not create the richer family of functions we need. A nonlinear activation changes this.

The rectified linear unit, or ReLU, is:

For $[-1,0.1,3.8]$, ReLU produces $[0,0.1,3.8]$. Positive values pass through; negative values become zero. Modern Transformer feed-forward blocks often use smoother or gated alternatives, but ReLU is useful for understanding the principle.

Another nonlinear function is the sigmoid:

For $z=0$, sigmoid gives 0.5; for $z=2$, approximately 0.881. A value between 0 and 1 is not automatically a calibrated probability of a real-world claim.

Interactive lab

Calculate a neural-network forward pass

Input x₁

2

Input x₂

1

ActivationReLUSigmoidIdentity (no nonlinearity)

First layerW = [[0.5, −1], [1, 2]]
b = [0.1, −0.2]

Output layerW₂ = [2, −0.5]
b₂ = 0.3

Change an input until a first-layer value crosses zero. ReLU then switches that unit between zero and its positive value.

3.4 Composing layers

A small two-layer network can be written:

Using the ReLU output $\mathbf{h}=[0.1,3.8]^T$, choose $W_2=[2,-0.5]$ and $b_2=0.3$. The result is:

The first layer builds an intermediate representation. The second combines it. The final value could be an intermediate score; additional operations would be needed to turn a vocabulary's scores into next-token probabilities.

3.5 Counting parameters

A dense layer with $d$ inputs and $m$ outputs contains $md$ weights and $m$ biases, for a total of $m(d+1)$ parameters. Our first layer has 4 weights plus 2 biases, or 6 parameters. The second has 2 weights and 1 bias. Together the network has 9 parameters.

The same ideas extend to networks with many layers and enormous numbers of parameters. Scale changes the computational requirements, while the basic operations remain recognisable.

3.6 A forward pass is not learning

A forward pass applies the current parameter values to the input. It produces representations and predictions. Learning requires an additional process: evaluate a loss, calculate derivatives, and update parameters. That is the subject of Lesson 5.

In a Transformer, a feed-forward block applies learned nonlinear transformations separately at each token position. Attention adds a way to combine information across positions. The two mechanisms play complementary roles.

Check your understanding

Lesson 3 checkpoint

Why add a nonlinear activation between affine layers?

A. It makes all outputs probabilitiesB. It removes the need for learned weightsC. It prevents the whole stack from collapsing into a single affine transformationD. It guarantees a correct answer

Check answer

The central insight from Lesson 3

Matrix multiplication combines features, biases shift values, and nonlinear functions let a network represent richer transformations. Next we will use learned matrices to calculate queries, keys, and values and build attention.

In Lesson 3, we learned how a neural network transforms vectors using matrix multiplication, weights, biases, and nonlinear functions.

We now come to the innovation that made modern Large Language Models possible: the attention mechanism.

This is arguably the most important lesson in our course.

The central question is deceptively simple:

How can the representation of one word be influenced by other words in the same sentence?

We'll work through the mathematics using small vectors, then see how the Transformer combines these operations into a powerful language-processing architecture.

4.1 The problem attention solves

Consider the sentence:

The pianist played the piano because it had just been tuned.

What does it refer to?

Almost certainly the piano, not the pianist.

Now consider:

The pianist played the piano because he had just finished rehearsing.

Here, he refers to the pianist.

To interpret either sentence correctly, the model needs to establish relationships between different tokens.

The word it cannot be understood adequately in isolation.

Its meaning depends on the preceding context.

A conventional feed-forward network transforms its input vector, but does not, by itself, decide which other token positions to consult.

Attention provides a mechanism for doing precisely that.

Conceptual illustration only. Attention weights do not necessarily correspond directly to grammatical reference.

The remarkable feature of attention is that these relationships are not programmed explicitly.

The network learns how to calculate them.

4.2 Three vectors: Query, Key, and Value

The attention mechanism gives each token three derived vectors:

• Query (Q): What information is this position looking for?
• Key (K): What characteristics does this position offer for matching?
• Value (V): What information can this position contribute?

These are useful interpretations, not literal descriptions of what individual vectors consciously represent.

A helpful analogy is a library search.

Suppose you're looking for books about jazz harmony.

Your query describes what you want.

Each book has an index entry, analogous to a key.

If the index entry matches your query, you retrieve information from the book, analogous to its value.

Attention works in a mathematically similar way.

But rather than retrieving one item, it usually constructs a weighted combination of information from multiple positions.

4.3 Where do Q, K, and V come from?

Remember the token embeddings from Lesson 2.

Suppose a token is represented by:

The model contains three learned weight matrices:

It calculates:

These are ordinary matrix multiplications, exactly as in Lesson 3.

For example, suppose:

Then:

The same token representation is multiplied by different matrices to produce its key and value.

The matrices are learned during training.

Consequently, the model can learn different numerical transformations for matching information and transmitting information.

4.4 Calculating an attention score

Let's use a very small example.

Suppose we have three tokens:

jazz piano music

For simplicity, imagine that the query for the current position is:

And the keys for the three positions are:

We compare the query with each key using the dot product.

For the first:

For the second:

For the third:

Our raw attention scores are:

The second position has the highest score.

But these are not yet probabilities or normalised weights.

We need another operation.

4.5 Why attention scores are scaled

Modern Transformers typically use scaled dot-product attention.

The raw score is divided by the square root of the key dimension:

Here $d_k$ is the number of components in the key vectors.

In our example, $d_k=2$.

So the scores become:

Approximately:

Why divide by $\sqrt{d_k}$?

If the components of queries and keys have roughly unit variance and behave approximately independently, their dot products tend to have a variance proportional to $d_k$.

Dividing by $\sqrt{d_k}$ helps keep the scores within a numerically manageable range.

Without scaling, very large dot products can cause the next operation, softmax, to become excessively concentrated on one position.

4.6 Softmax: turning scores into attention weights

We now have three scores:

We want to convert these into nonnegative weights that add up to 1.

The function used is called softmax:

Here $e\approx2.71828$, the base of natural logarithms.

Let's calculate it.

First, exponentiate the scores:

Add them:

Then divide each by the total:

So our attention weights are approximately:

The second position receives about half the attention weight.

These are normalised mixing coefficients, not probabilities that a token is the correct grammatical referent.

Experiment with softmax

Interactive lab

Softmax and a weighted sum

Teaching temperature

1

Weighted output vector[0.993, 1.255]

Scores are [1, 2, 1] / √2. Values are [1, 0], [0, 2], and [3, 1]. Temperature divides scores here for teaching; it is distinct from the decoding control in Lesson 6.

Notice that softmax doesn't simply select the highest score.

It distributes weight across the available positions.

This allows the network to combine information from several tokens.

4.7 Values: what information gets combined?

We have calculated attention weights, but we haven't yet used the value vectors.

Suppose the three positions have values:

The attention output is the weighted sum:

Substituting:

For the first component:

For the second:

Therefore:

This is the crucial result.

The attention mechanism has constructed a new vector by combining information from multiple token positions.

The relative contributions depend on the query-key scores.

And the matrices used to calculate queries, keys, and values are all learned during training.

4.8 Attention as a matrix operation

So far, we've calculated attention for one query.

But a Transformer processes many token positions.

We can collect all the queries into a matrix $Q$, all the keys into $K$, and all the values into $V$.

The entire operation becomes:

This is one of the central equations of modern AI.

Let's unpack it.

First:

calculates dot products between queries and keys.

The superscript $T$ means transpose, which exchanges rows and columns.

Next:

scales those scores.

Then:

converts each row of scores into attention weights.

Finally, multiplying by $V$ produces weighted combinations of value vectors.

We can visualise the attention weights as a matrix.

Interactive lab

Inspect a causal attention matrix

Choose a query row. Within that row, weights sum to 1; future positions are masked.

Query position1 · The2 · pianist3 · played4 · piano

These are fictional weights, chosen to illustrate masking. They are not measured attention from a trained model.

The matrix representation is important because modern hardware, particularly GPUs, can perform large matrix multiplications extremely efficiently.

That computational efficiency is one reason Transformers scale so well.

4.9 Causal masking: why the model cannot look into the future

When an autoregressive LLM generates text, it predicts the next token from the preceding tokens.

During training, however, the model can process an entire training sequence in parallel.

How do we stop it from cheating by attending to future tokens?

We use a causal mask.

Suppose the sequence contains four tokens.

The allowed attention pattern is:

The first position can attend only to itself.

The second can attend to the first and second.

The third can attend to the first three.

And so on.

Mathematically, the model adds a very large negative value, ideally represented as $-\infty$, to the scores of forbidden positions before applying softmax.

Since:

those positions receive zero attention weight.

This ensures that a token's representation depends only on permitted preceding context.

It also allows training computations for many positions to be performed in parallel.

4.10 Multi-head attention

So far, we've described one attention mechanism.

But modern Transformers typically use multiple attention heads.

Each head has its own learned query, key, and value projections.

This allows different heads to calculate different patterns of relationships.

For example, in:

The pianist who accompanied the singer played beautifully.

One head might develop a pattern useful for linking pianist with played.

Another might attend to nearby words.

Another might track punctuation or sentence boundaries.

These are possible learned behaviours, not fixed roles assigned to particular heads.

Mathematically, each head computes:

The outputs are then concatenated and transformed by another learned matrix:

Here $W_O$ is an output projection matrix.

The result is a new representation for each token position.

4.11 Why position matters

Consider two sentences:

The dog chased the cat.

The cat chased the dog.

They contain almost the same tokens, but describe opposite events.

If a model only considered an unordered collection of token embeddings, it could not reliably distinguish them.

The Transformer therefore needs positional information.

The original Transformer architecture used sinusoidal positional encodings.

For example:

Here:

• $pos$ is the token position.
• $i$ indexes pairs of embedding dimensions.
• $d$ is the embedding dimension.

These functions create different numerical patterns for different positions.

Many modern LLMs instead use rotary positional embeddings, or RoPE.

RoPE rotates components of query and key vectors by position-dependent angles.

This allows attention scores to reflect relative positional relationships.

The mathematics is more involved, but the underlying purpose is straightforward:

The model needs to know not only which tokens are present, but where they occur relative to one another.

4.12 Putting a Transformer layer together

We can now assemble the main components.

A simplified Transformer layer works as follows:

Conceptual diagram. Implementations vary in normalisation order, residual connections, and other details.

A large language model stacks many such layers.

Each layer receives the representations produced by the preceding layer.

As information passes through the network, token representations can become increasingly sensitive to complex relationships in the context.

4.13 Why attention is so powerful

There are three particularly important consequences.

First, attention is context-dependent. The weights are calculated from the current input, rather than being fixed for every sentence.

Second, attention can connect distant positions. A token near the end of a long passage can incorporate information from a token much earlier in the passage, provided that position is within the context window and permitted by the attention mask.

Third, attention is differentiable. The calculations use mathematical operations for which derivatives can be computed. That means the query, key, value, and output matrices can all be learned through gradient-based training.

This last point is especially important.

Nobody has to tell the network explicitly which words should attend to which other words.

Training can discover attention patterns that improve next-token prediction.

There are limitations, of course. Attention weights are not guaranteed to provide a faithful explanation of a model's reasoning, and long-context performance can deteriorate even when earlier tokens are technically accessible.

But the architecture provides a remarkably flexible mechanism for combining information.

Check your understanding

Lesson 4 checkpoint

Suppose a query produces scaled attention scores [1, 2, 1]. What does softmax do?

A. Selects the second position and discards the othersB. Produces positive weights summing to 1, with the second position receiving the greatest weightC. Multiplies each score by its token IDD. Changes the scores directly into token embeddings

Check answer

The central insight from Lesson 4

We have now seen the essential mathematical mechanism behind the Transformer:

It is worth remembering what this equation accomplishes.

The model calculates relationships between token positions, converts those relationships into weights, and uses the weights to combine information from across the context.

All of this happens through matrix multiplication and other differentiable mathematical operations.

We now understand the main ingredients of an LLM's forward computation.

But one enormous question remains:

How can training discover the billions of numerical weights that make these computations useful?

That is what we'll tackle in Lesson 5, where we'll revisit calculus, derivatives, and optimisation to understand how an LLM actually learns.

In the first four lessons, we discovered how an LLM performs calculations:

• Text is converted into tokens and embeddings.
• Matrices transform the embeddings into new numerical representations.
• Attention allows tokens to incorporate information from other positions.
• Successive Transformer layers produce representations used to predict the next token.

But we have so far avoided one fundamental question:

Where do the billions of weights in those matrices come from?

The answer is that the model learns them by repeatedly making predictions, measuring its errors, and adjusting its weights to reduce those errors.

The mathematics rests on calculus, particularly derivatives and the chain rule.

This lesson will connect those familiar ideas to the training of an LLM.

5.1 Learning as an optimisation problem

Imagine a tiny language model that has encountered the sentence:

The pianist played the ...

Its task is to predict the next token.

Suppose the correct token in its training example is piano, but the model assigns these probabilities:

The model has made a poor prediction. It assigns only 20% probability to the actual next token.

We need a mathematical way of measuring how poor that prediction is.

This is the purpose of a loss function.

Training aims to minimise the average loss over many examples.

In mathematical notation:

Here:

• $\theta$ represents all the model's adjustable parameters.
• $L(\theta)$ is the loss calculated using those parameters.
• $\arg\min$ means the parameter values that minimise the loss.

In a large LLM, $\theta$ can contain billions of numbers.

We cannot search through all possible combinations of those numbers.

Instead, we use derivatives to determine how to adjust them.

But first, we need to define the loss.

5.2 Cross-entropy: measuring prediction errors

For next-token prediction, the standard training loss is cross-entropy.

If the correct next token has predicted probability $p$, the loss for that example is:

Here $\ln$ is the natural logarithm.

Why use a logarithm?

Consider three possibilities.

Notice that the loss increases sharply as the probability of the correct answer approaches zero.

A model that assigns 1% probability to the correct token is penalised much more heavily than one that assigns 50%.

This is desirable because confidently wrong predictions should incur substantial loss.

Explore cross-entropy

Interactive lab

Cross-entropy for one correct token

Probability of the correct token

0.2

L = −ln(p)1.609

At p = 1, loss is 0. As p approaches 0 from above, loss grows without bound. This slider stops at 0.01.

For a vocabulary with many possible tokens, the general cross-entropy formula is:

where $y_i$ represents the target distribution and $p_i$ the model's predicted distribution.

For ordinary next-token training, the target is usually one-hot: it assigns 1 to the actual next token and 0 to every other token.

The formula therefore reduces to $-\ln(p)$ for the correct token.

Across a training sequence, we average or sum the losses at the predicted token positions.

5.3 A reminder: what does a derivative tell us?

Suppose we have a simple function:

We want to find the value of $w$ that minimises $L$.

From elementary calculus:

The derivative tells us how rapidly the loss changes when we change $w$.

At $w=5$:

The derivative is positive. Increasing $w$ increases the loss locally, so moving $w$ downwards should help.

At $w=1$:

The derivative is negative. Increasing $w$ should reduce the loss.

At $w=3$:

We've reached the minimum.

Interactive lab

See the derivative as a slope

Weight w

5

L(w) = (w − 3)²4.000

dL/dw = 2(w − 3)4.000

The dashed line is the tangent at the chosen point. Its slope is negative to the left of 3, zero at 3, and positive to the right.

This is the basic idea behind training a neural network.

Except that instead of one variable $w$, we have billions of parameters.

5.4 Gradient descent: adjusting the weights

The simplest optimisation method is gradient descent.

Its update rule is:

The Greek letter $\eta$, pronounced eta, is the learning rate.

It controls the size of each adjustment.

Let's use:

Start with $w=5$ and choose learning rate $\eta=0.1$.

The derivative is:

So:

The loss changes from:

to:

We've reduced the loss.

Repeating the process:

The weight gradually approaches 3.

Try gradient descent yourself

Interactive lab

Take gradient descent steps

Learning rate η

0.1

Steps0

Weight w5.000

Loss4.000

Take one stepTake five stepsReset

Step	Weight	Loss

For this quadratic, the error is multiplied by (1 − 2η) on each step. Rates below 1 converge; η = 1 oscillates; rates above 1 diverge. Other loss functions have different stability limits.

In a real LLM, the loss depends on many parameters.

We therefore use partial derivatives.

For example:

The gradient is the vector:

The symbol $\partial$ means a partial derivative: how the loss changes with respect to one parameter, holding the others fixed.

The update becomes:

This is gradient descent in many dimensions.

Actual LLM training typically uses sophisticated variants, such as AdamW, together with learning-rate schedules and other numerical techniques.

But the central idea remains the same.

5.5 Backpropagation: how the model calculates the derivatives

We now reach the most important mathematical idea in training neural networks.

Suppose our network performs two operations:

followed by:

And suppose the loss is:

where $t$ is the target.

We want to know:

But $L$ depends on $w$ indirectly, through $z$ and $y$.

This is exactly the situation addressed by the chain rule from calculus.

Let's work through the numbers.

Choose:

First, perform the forward calculation:

Now calculate the derivatives backwards.

Step 1: Loss with respect to output

Step 2: Output with respect to intermediate value

Step 3: Intermediate value with respect to weight

Multiply them:

That is the gradient for the weight.

It tells us that a small increase in $w$ would increase the loss very rapidly at this point.

So gradient descent will reduce $w$.

For example, using a small learning rate of $0.001$:

The new prediction becomes:

Much closer to the target of 20.

The loss falls from 256 to approximately 0.0053.

Using the unrounded prediction 19.927296 gives a new loss of 0.00528587.

This is an unusually dramatic improvement because we've chosen a simple example with a convenient learning rate. Real optimisation is often much less smooth.

Visualising backpropagation

Interactive lab

Follow the chain rule

Weight w (x = 2, target = 20)

3

Forward

Backward

Update with η = 0.001

This is the exact worked graph z = wx, y = z², L = (y − 20)². Starting at w = 3 gives gradient 768 and the next weight 2.232.

This process is called backpropagation.

It is not a separate mysterious learning algorithm.

It is an efficient application of the chain rule to a computational graph.

The model calculates the output going forwards, then calculates derivatives going backwards.

5.6 Why backpropagation is computationally efficient

Imagine a network containing a billion parameters.

A naïve approach would be to change each parameter slightly, recalculate the entire loss, and observe what happened.

That would require an enormous number of forward calculations.

Backpropagation avoids this.

It reuses intermediate results and propagates derivatives backwards through the computational graph.

For a scalar loss, reverse-mode automatic differentiation can calculate gradients for all parameters with a computational cost of the same general order as a small number of forward passes, rather than requiring one complete evaluation per parameter.

This efficiency is what makes training very large neural networks practical.

5.7 How the embedding vectors are learned

Let's connect this back to Lesson 2.

Remember the embedding matrix:

Each entry is a trainable parameter.

Suppose the embedding for piano is:

After a forward pass, backpropagation might calculate the gradient:

With learning rate $0.1$, the update is:

So:

The token embedding has changed slightly.

Across enormous numbers of training examples, these small adjustments help embeddings acquire useful relationships.

The numbers in this example are invented, but the principle is exactly how learned embedding parameters can be updated.

5.8 How the attention matrices are learned

Now consider the attention mechanism from Lesson 4.

Recall:

The matrices $W_Q$, $W_K$, and $W_V$ contain trainable weights.

During training:

1. The model uses these matrices to calculate attention.
2. Attention produces contextual token representations.
3. The model uses those representations to predict the next token.
4. Cross-entropy measures the prediction error.
5. Backpropagation calculates how the loss depends on each attention weight.
6. The optimiser adjusts those weights.

For example, a particular weight in $W_Q$ might be updated from:

to:

That tiny adjustment changes the query vectors produced by the model.

Which changes the attention scores.

Which changes the attention weights.

Which changes the contextual representations.

Which changes the next-token probabilities.

This illustrates how training can gradually discover useful attention patterns.

The same process updates the key, value, and output projection matrices.

5.9 How the feed-forward layers are learned

The feed-forward networks from Lesson 3 are trained in the same way.

A simplified feed-forward layer might calculate:

followed by:

Backpropagation calculates gradients for:

The optimiser then updates those parameters.

Importantly, the embeddings, attention matrices, and feed-forward weights are not normally trained independently.

They are jointly optimised to reduce the same overall prediction loss.

A change in one part of the network can alter what representations are useful elsewhere.

This joint optimisation is one reason large neural networks can develop complex internal structures.

5.10 Training on many examples: stochastic gradient descent

So far, we've considered one training example at a time.

But real LLMs train on enormous datasets.

Suppose we have a collection of $N$ token predictions.

The overall objective might be:

Here $w_{<i}$ means the tokens preceding position $i$.

Calculating the exact gradient over the entire dataset before every update would be impractical.

Instead, training uses mini-batches.

A mini-batch contains a subset of training examples.

The model calculates the average loss over that batch, computes its gradient, and updates the parameters.

This is a form of stochastic optimisation.

The gradient from a mini-batch is an estimate of the gradient of the overall training objective.

Over many updates, the model can progressively improve.

5.11 What happens during a real LLM training step?

Let's put everything together.

Training text

A batch of token sequences

Forward pass

Embeddings → attention → feed-forward layers → next-token probabilities

Cross-entropy loss

Measure how much probability the model assigned to the actual next tokens

Backpropagation

Calculate gradients for the trainable parameters

Optimiser update

Adjust weights to reduce the loss

Repeat with another mini-batch

This cycle is repeated an enormous number of times.

Training can involve trillions of tokens and very large amounts of computation.

The model gradually develops parameter values that enable it to predict text across many domains.

5.12 Does the model simply memorise its training data?

This is an important question.

The answer is that models can both memorise and generalise.

Memorisation means retaining information about particular training examples, sometimes closely enough to reproduce them.

Generalisation means learning patterns that are useful for inputs not encountered during training.

For example, a model might learn how to multiply numbers from many worked examples and then apply that procedure to unfamiliar numbers.

Or it might learn relationships between programming constructs and use them to generate code that was never present in the training data.

But there is no guarantee that a learned procedure will always be applied correctly.

A model can learn shortcuts, spurious correlations, or incomplete rules.

That is why high performance on familiar-looking tasks does not automatically establish reliability on unfamiliar or safety-critical problems.

5.13 Pretraining versus post-training

One final distinction is important.

What we've described so far is primarily pretraining.

During pretraining, the model learns to predict tokens from large collections of text and other suitable data.

But a model trained only this way is not necessarily a good conversational assistant.

It may continue text rather than follow instructions, or produce outputs that are inappropriate for a user's goals.

Additional training stages can improve its behaviour.

These may include supervised instruction training, where the model learns from examples of desired responses, and preference-based or reinforcement-learning methods that encourage useful, accurate, and appropriate behaviour.

The exact methods vary between model families.

The essential point is that next-token prediction provides the broad foundation, while post-training helps shape how the model uses its capabilities.

Check your understanding

Lesson 5 checkpoint

A weight is 4, its loss derivative is +6, and the learning rate is 0.1. What is the updated weight?

A. 4.6B. 3.4C. 0.6D. 4.0

Check answer

The central insight from Lesson 5

We now have the essential mathematical explanation of how an LLM learns.

It begins with adjustable parameters, uses them to make predictions, measures the resulting errors, and calculates how each parameter contributes to those errors.

The key equations are:

The extraordinary thing is that this relatively simple optimisation principle, applied to a very large Transformer and an enormous amount of data, can produce a system capable of generating explanations, writing software, translating languages, and solving many unfamiliar problems.

But the training process is only half the story.

In Lesson 6, we'll examine what happens when you actually ask an LLM a question: how the trained model produces next-token probabilities, how tokens are selected, why answers can vary, and how the model handles a growing conversation.

That will complete our account of the basic computational machinery, leaving us ready for the deeper question in Lesson 7: why these calculations can produce behaviour that looks like understanding and reasoning.

We now understand how an LLM is constructed and trained. In this lesson, we'll examine what happens when you actually ask it a question.

For example:

Explain why the sky is blue.

The model doesn't retrieve a complete, prewritten answer. Instead, it calculates a probability distribution over possible next tokens, selects one, adds it to the context, and repeats.

We'll follow that process from beginning to end, covering seven ideas:

1. Logits: the raw scores produced by the neural network.
2. Softmax: converting scores into probabilities.
3. Temperature: controlling how concentrated the probabilities are.
4. Top-k and top-p: restricting which tokens may be selected.
5. Autoregressive generation: constructing an answer token by token.
6. Context windows: what the model can access during generation.
7. KV caching: how generation is made computationally efficient.

We'll finish by explaining why the same question can produce different answers, even when the model's weights haven't changed.

6.1 From the Transformer to logits

Recall the pipeline from our earlier lessons:

Input text

"The pianist played the"

Tokenisation and embeddings

Transformer layers

Attention and feed-forward calculations

Final hidden vector

Logits for every possible next token

The final hidden vector is transformed into a set of numbers, one for each token in the model's vocabulary.

These numbers are called logits.

Suppose our toy model has just four possible next tokens.

A logit is not a probability.

It is an unnormalised numerical score.

It can be positive, negative, or zero. What matters is the relative difference between logits.

How are logits calculated?

Suppose the final hidden vector is:

And the output projection matrix is:

Then:

These are our four logits.

Real LLMs use much larger matrices. The output projection may also share parameters with the input embedding matrix, a technique called weight tying.

But the underlying mathematics is exactly the matrix multiplication we studied in Lesson 3.

6.2 Softmax converts logits into probabilities

We encountered softmax in Lesson 4, when calculating attention weights.

The same mathematical function is used to turn output logits into next-token probabilities.

For our logits:

we first exponentiate:

Approximately:

Their sum is:

Dividing each by that total gives:

Notice that the highest-scoring token has the greatest probability, but the others remain possible.

One useful mathematical property is that adding the same constant to every logit doesn't change the probabilities:

This is why implementations often subtract the largest logit before exponentiating. It improves numerical stability without changing the result.

6.3 Temperature: how adventurous should the model be?

Suppose we want the model to be more predictable, or alternatively to consider less likely continuations more often.

We can change the distribution using a parameter called temperature.

The formula becomes:

where $T$ is a positive number.

At $T=1$, the original probabilities are unchanged.

At a lower temperature, differences between logits become more pronounced.

At a higher temperature, the distribution becomes more even.

Interactive lab

Change decoding temperature

Temperature T

1

T = 0.5T = 1T = 2

The logits remain [3, 2, 1, 0]. The learned model is unchanged; only the selection distribution changes.

For comparison:

Temperature does not change the model's learned weights or knowledge. It changes how the output scores are converted into a distribution for selection.

A lower temperature does not guarantee factual accuracy. It mainly reduces randomness.

At the mathematical limit $T\to0^+$, the distribution concentrates on the highest logit. In practice, deterministic decoding is often implemented directly by choosing the highest-scoring token.

6.4 How does the model select a token?

Once the model has calculated probabilities, it needs to choose the next token.

The simplest method is greedy decoding.

This means always selecting the token with the highest probability.

In our example, greedy decoding always chooses piano.

But another approach is sampling.

The model draws a token at random according to the probability distribution.

With the original distribution, piano would be selected approximately 64% of the time, violin 24%, and so on, across many independent samples.

This means that the same input can produce different continuations.

Try sampling from the distribution

Interactive lab

Sample a next token

The pianist played the …

Sample onceSample 1,000 timesReset counts

Fixed temperature 1; each draw uses the four-token distribution from this lesson. Empirical frequencies fluctuate around the theoretical probabilities.

Notice that sampling does not mean choosing arbitrarily.

More likely tokens are selected more often.

But less likely tokens are not necessarily excluded.

This can be useful for creative writing, brainstorming, and generating varied responses.

It can also produce undesirable or incoherent continuations if very unlikely tokens are selected.

That brings us to two further techniques.

6.5 Top-k sampling

Top-k sampling restricts the model to the $k$ highest-scoring tokens.

Suppose $k=2$.

Our original probabilities were:

Top-k keeps only piano and violin.

Their probabilities are renormalised so they add up to 1.

The new probabilities are approximately:

The other two tokens receive probability zero.

This prevents the model from selecting very low-ranked tokens.

But the value of $k$ is fixed, regardless of how concentrated or diffuse the original probability distribution is.

That limitation motivates another approach.

6.6 Top-p sampling, also called nucleus sampling

Instead of keeping a fixed number of tokens, top-p sampling keeps the smallest set of highest-probability tokens whose cumulative probability reaches or exceeds a threshold $p$.

For example, with $p=0.9$:

• piano contributes 64.4%.
• Adding violin brings the total to 88.1%.
• Adding guitar brings it to 96.8%.

So the retained set is:

piano, violin, guitar.

drums is excluded.

The remaining probabilities are renormalised.

Unlike top-k, the number of retained tokens can vary according to the shape of the probability distribution.

Compare top-k and top-p

Interactive lab

Compare top-k and top-p

FilterTop-kTop-p (nucleus)

Keep k tokens

2

The original probabilities come from logits [3, 2, 1, 0] at temperature 1. Retain the ranked prefix, then renormalise. Top-p includes the token that crosses the threshold.

In practice, temperature, top-k, and top-p may be combined, and different systems use different decoding strategies.

They are not compulsory features of every LLM deployment.

6.7 Autoregressive generation: constructing the answer

We can now describe the entire generation process.

Suppose the prompt is:

The pianist played the

The model predicts the next token.

Perhaps it selects piano.

The new context becomes:

The pianist played the piano

It then predicts another token.

Perhaps it selects beautifully.

The context becomes:

The pianist played the piano beautifully

The model repeats this process until it produces an end-of-turn token, reaches a length limit, or another stopping condition is met.

This is called autoregressive generation.

Mathematically, the probability of a complete sequence is:

We encountered this chain rule in Lesson 1.

The important point is that each new token becomes part of the context used to predict subsequent tokens.

A miniature generation tree

Illustrative generation paths. The second-stage probabilities are not shown because they would depend on which first-stage token was selected.

An important consequence follows.

If the model makes a different choice early in its response, that changes the context for all subsequent predictions.

A small initial difference can therefore lead to a substantially different answer.

6.8 What is a context window?

An LLM doesn't necessarily have access to every word ever written, or even every word in a very long conversation.

It operates within a context window.

The context window is the amount of tokenised information the model can process for a given generation.

Depending on the model and deployment, this might contain thousands, hundreds of thousands, or more tokens.

The context may include system instructions, user messages, earlier assistant responses, retrieved documents, tool results, and the tokens already generated in the current answer.

The context window is not the same as the model's trained knowledge.

Trained knowledge is encoded imperfectly in the model's parameters.

Context is the information supplied to the model for the current computation.

An analogy

Think of the model's trained weights as something like long-term acquired competence.

The context window is more like the material currently available on a desk.

The analogy is imperfect, but useful.

A model may have learned how to reason about contracts during training.

If you provide a particular contract in the context, it can apply those learned capabilities to that document.

However, if a relevant clause falls outside the available context, the model may no longer have access to its exact wording.

A larger context window allows more information to be included, but does not guarantee that the model will use every part of it reliably.

Interactive lab

Budget a context window

Toy context capacity (tokens)

64

Input tokens (all instructions and supplied text)

40

Reserved output tokens

16

This simplified budget treats input plus reserved output as sharing one capacity. Actual systems can have separate output limits and their own truncation or summarisation policies.

6.9 Why generation would be expensive without caching

Let's examine the computational problem.

Suppose the model has already processed 1,000 tokens.

It generates token 1,001.

To generate token 1,002, it needs to take account of the preceding 1,001 tokens.

A naïve implementation might recompute all the Transformer calculations for the entire sequence every time another token is generated.

That would waste enormous amounts of computation.

The solution is KV caching.

KV stands for Key-Value.

Recall from Lesson 4 that attention uses queries, keys, and values.

For each Transformer layer, the model can store the key and value representations calculated for earlier token positions.

When a new token arrives, it computes the new token's query, key, and value.

It then compares the new query with the cached keys and uses the corresponding values to calculate attention.

The previous keys and values do not need to be recomputed in a standard causal Transformer, because earlier positions cannot attend to future tokens.

KV caching substantially reduces repeated computation during autoregressive generation.

However, the cache consumes memory, and attention to a growing cache still requires work. KV caching doesn't make the cost of processing additional context disappear.

Modern systems may also use grouped-query attention, multi-query attention, cache quantisation, or other techniques to reduce memory requirements.

Interactive lab

Count repeated projection work

Prompt tokens

32

Generation steps

16

Full-prefix token projections632

With cached prefixes47

A simplified count of token positions processed by projection/feed-forward layers: without caching sum N + (N+1) + … + (N+M−1); with caching N + (M−1). It is not a count of total attention FLOPs or elapsed time. Growing-cache attention still costs work.

6.10 Why the same question can produce different answers

We can now answer this question precisely.

There are several reasons.

First, stochastic sampling. If the model samples from a probability distribution, different tokens may be selected on different runs.

Second, early differences propagate. Once one token differs, subsequent predictions are conditioned on different text.

Third, the effective context may differ. Conversation history, instructions, retrieved information, and tool results can change the input.

Fourth, the model or serving configuration may differ. Updates to model weights, decoding settings, or system configuration can affect outputs.

Fifth, numerical computation is not always perfectly deterministic. Hardware and parallel computation can introduce small differences, although these do not necessarily affect the selected tokens.

Even greedy decoding does not guarantee identical outputs across every possible deployment and computational environment.

But under a fixed model, fixed context, fixed implementation, and deterministic computation, greedy decoding should produce the same continuation.

6.11 Does the model plan the whole answer before writing it?

This is a subtle question.

The basic autoregressive mechanism generates one token at a time.

It does not ordinarily calculate a complete final answer and then simply print it.

However, that does not mean the model is incapable of planning.

Its internal representations can encode information about the structure of a response, likely future developments, and relationships between ideas.

Some systems also use additional inference-time computation, intermediate reasoning steps, or other mechanisms before producing the final response.

The important distinction is between the fundamental token-generation mechanism and the broader computational process that may surround it.

We should therefore avoid two oversimplifications:

• An LLM is not merely selecting random words without structure.
• An LLM does not necessarily construct a complete, verified answer before beginning to generate text.

The actual behaviour depends on the model architecture, training, and inference configuration.

Check your understanding

Lesson 6 checkpoint

Probabilities are A: 0.50, B: 0.30, C: 0.15, D: 0.05. Which tokens does top-p retain when p = 0.75?

A. Only AB. A and BC. A, B, and CD. All four

Check answer

The central insight from Lesson 6

We now understand the complete basic generation mechanism.

The Transformer produces logits, softmax converts them into probabilities, and a decoding strategy selects the next token.

The selected token is appended to the context, and the process repeats.

KV caching makes this efficient by reusing previously calculated attention keys and values.

This completes our mathematical account of how a conventional autoregressive LLM is trained and generates text.

But it leaves us with the deepest question of all.

Why should a system trained to predict tokens acquire the ability to reason about mathematics, explain physical phenomena, write software, or discuss philosophical ideas?

And perhaps more importantly, how can we distinguish genuine competence from plausible-sounding but unreliable output?

Those are the questions we'll explore in Lesson 7.

We've reached the most interesting question in our course.

In Lessons 1–6, we established that an LLM is trained to predict tokens using a very large mathematical function. Its weights are adjusted through gradient descent, and it generates responses by repeatedly calculating probabilities over possible continuations.

But this raises a puzzle:

Why should predicting the next token produce anything resembling reasoning or understanding?

After all, predicting words sounds rather different from understanding physics, proving theorems, or designing software.

To answer this, we'll explore five connected ideas:

1. Why next-token prediction creates pressure to learn relationships about the world.
2. How reasoning can emerge from learned computations.
3. The distinction between memorisation and generalisation.
4. Why hallucinations occur, even in highly capable models.
5. What we can, and cannot, conclude about genuine understanding.

We'll use mathematical examples throughout, and finish by connecting these ideas to the design of trustworthy AI systems.

7.1 Predicting language can require modelling the world

Let's begin with a simple physical situation.

A stone is dropped from rest from a height of 20 metres. Ignoring air resistance, it takes approximately ___ seconds to reach the ground.

A model that knows nothing about physics could try to predict the missing number using statistical associations.

But a model that has learned the relationship between distance, acceleration, and time can make a much better prediction.

Recall the equation:

Rearranging:

For $h=20$ metres and $g=9.81\ \mathrm{m/s^2}$:

The correct continuation is therefore approximately 2.02 seconds.

The important point is that the numerical answer depends on a relationship between physical quantities.

If the model learns that relationship, it can predict answers to many related questions.

Interactive lab

Apply a physical relationship

Drop height (metres)

20

Time to fall2.02 s

Impact speed19.81 m/s

Calculated directly from idealised Newtonian free fall with g = 9.81 m/s² and initial speed zero. This is a deterministic formula, not an LLM prediction.

This is the first key insight.

To predict language accurately, a model can benefit from learning the structure of the things language describes.

That structure may include physical laws, grammatical rules, mathematical relationships, social conventions, programming semantics, and causal relationships.

A model is not explicitly instructed to learn all of these during ordinary next-token pretraining.

But learning useful representations of them can reduce prediction loss.

7.2 Why this follows from probability theory

Let's formalise the argument.

Suppose $X$ represents the text preceding the next token, and $Y$ represents that next token.

The model attempts to estimate:

Now suppose there is some underlying fact or state of the world, $Z$, that helps explain the text.

For example, $Z$ might represent the physical state of a falling object.

The next token may depend on both the text and that underlying state:

If knowing $Z$ helps predict $Y$, then a model that can infer useful information about $Z$ from $X$ may achieve a lower expected loss.

We can express the theoretical benefit using conditional entropy.

The uncertainty about $Y$ given $X$ is:

The uncertainty given both $X$ and $Z$ is:

Their difference is:

This is called conditional mutual information.

It measures how much additional information $Z$ provides about $Y$, once $X$ is known.

If this quantity is positive, then $Z$ contains predictive information beyond what is available in $X$ alone.

There's an important qualification.

An LLM does not normally receive the true world state $Z$ directly. It has access to text, and perhaps other supplied inputs.

It must infer useful representations from those inputs.

And if $Z$ cannot be inferred from the available evidence, no amount of clever computation can reliably recover it.

This distinction will become important when we discuss hallucinations.

If a representation is a deterministic function of the available text, it cannot add information beyond that text. The conditional mutual information above describes the benefit of additional world-state evidence. Learning a representation can make information already present in the text more usable by a finite network; it does not evade the information limit.

7.3 What does an LLM actually learn about the world?

An LLM does not normally store a conventional database of facts.

Instead, information is distributed across the numerical parameters of the network.

Some learned representations may capture relatively simple associations:

• Paris is associated with France.
• A violin is a string instrument.
• A triangle has three sides.

Other representations may encode relationships that support more flexible computations.

For example, the model may learn patterns relevant to:

• Comparing quantities.
• Tracking entities through a paragraph.
• Applying mathematical transformations.
• Following programming-language syntax.
• Distinguishing cause from consequence.

But these representations are not guaranteed to correspond to neat, explicit concepts.

A model might represent a relationship through the combined activity of many neurons, attention heads, and layers.

Moreover, the same internal components can participate in different computations.

This is one reason interpreting the internal structure of an LLM remains an active research problem.

An important distinction

There are at least three different senses in which we might say a model "knows" something.

These are not equivalent.

A model may have strong general knowledge about weather systems while having no direct information about today's weather in a particular location.

It may understand how an organisation's policies usually work without knowing the organisation's current approved policy.

And it may be able to explain how to verify a calculation without actually having verified it.

7.4 How reasoning can emerge from next-token prediction

Let's consider a mathematical problem:

If $3x+5=20$, what is $x$?

A direct answer is:

But we can also express the solution as a sequence of intermediate steps:

Each step follows from the previous one.

During training, a model encounters many examples of mathematical arguments, proofs, explanations, algorithms, and problem-solving procedures.

Predicting such text can reward the model for learning patterns of valid transformations.

For example, it may learn that subtracting the same quantity from both sides of an equation preserves equality.

If it learns a sufficiently general representation of that operation, it can apply it to equations not encountered during training.

This is more interesting than memorising the answer to a particular problem.

It amounts to learning a procedure.

A simple computational interpretation

Suppose a problem-solving process consists of states:

A reasoning step transforms one state into another:

For the equation above:

The model may learn to produce token sequences that implement transformations resembling $F$.

But an LLM does not necessarily store $F$ as an explicit symbolic rule.

The computation may instead be distributed across its learned parameters and intermediate representations.

This is why an LLM can sometimes perform impressive reasoning without possessing a conventional symbolic theorem prover.

It is also why its reasoning can be inconsistent.

7.5 Does generating intermediate steps improve reasoning?

Often, yes.

Consider two approaches to a problem.

Approach A:

Question → Answer

Approach B:

Question → Intermediate step 1 → Intermediate step 2 → Answer

In the second approach, each intermediate step becomes part of the context for subsequent predictions.

This allows the model to perform a longer sequence of computations.

In effect, generating intermediate text can provide additional working space.

The difference can be substantial.

A problem requiring several transformations may be difficult to solve in one prediction but easier when decomposed into steps.

However, we must be careful.

A plausible-looking explanation is not proof that the model actually used that explanation to obtain its answer.

The generated steps may contain errors, omit important reasoning, or rationalise an answer reached by other means.

Also, some reasoning-oriented models use internal inference-time computation that is not fully represented by the final explanation shown to the user.

The reliable conclusion is that intermediate computation can improve problem-solving performance, not that every verbal explanation faithfully reveals the model's internal process.

7.6 Memorisation versus generalisation

This distinction is central to understanding LLM capability.

Suppose a model encounters the following examples during training:

There are two very different ways it might perform well on these examples.

Memorisation: Store associations between the specific inputs and outputs.

Generalisation: Learn a procedure that can calculate sums for new inputs.

The distinction becomes apparent when we test unfamiliar cases.

For example:

A system that has learned addition can solve this without having seen the exact expression before.

A system that has merely memorised the three examples cannot reliably do so.

A visual analogy

Interactive lab

A lookup table versus a reusable rule

Training examples: (1, 3), (2, 5), (3, 7).

Test input x

5

Exact lookupNot in table

Rule: y = 2x + 111

Many rules can fit three points. This demonstration shows how a rule can extend to new inputs; it does not establish how an actual LLM solved a task or prove that this rule is the right one.

Modern LLMs exhibit both behaviours.

They can memorise particular sequences, including some that appear in their training data.

They can also generalise to unfamiliar tasks, combinations of ideas, and examples.

The difficulty is determining which mechanism is responsible for a particular successful answer.

How can we test generalisation?

One approach is to construct examples that differ systematically from the training distribution.

For example, if a model performs well on two-digit addition, test three-digit, four-digit, and much longer numbers.

If it performs well on familiar programming problems, test new combinations of requirements.

If it can explain a scientific principle, test whether it can apply that principle to unfamiliar boundary conditions.

The strongest evidence of generalisation comes from successful performance on genuinely novel, well-controlled tasks.

But novelty is difficult to establish when training datasets are enormous and not fully documented.

7.7 Why do LLMs hallucinate?

A hallucination occurs when a model produces information that is false, unsupported, or fabricated, often in a confident and plausible form.

Why does this happen?

The answer follows from the training objective.

Recall that the model is trained to minimise:

This objective rewards predicting tokens that occur in the training data.

It does not directly require every generated statement to be independently verified against reality.

A model can therefore learn the statistical structure of authoritative-sounding explanations without always having the evidence needed to justify them.

A simple example

Suppose you ask:

What did the fictional physicist Eleanor Voss discover in 1937?

If no such person exists, the correct response would be to say that the premise cannot be verified.

But a language model might generate:

Eleanor Voss discovered the Voss effect in 1937, establishing a relationship between magnetic fields and crystal resonance.

Why?

Because the question resembles many genuine historical questions.

The model may have learned that questions of this form are often followed by descriptions of discoveries, dates, and named effects.

It can produce a linguistically plausible continuation without possessing evidence that the person or discovery exists.

The resulting statement may be coherent but false.

7.8 A probabilistic explanation of hallucination

Let's formalise the problem.

Suppose the model has to choose between two responses:

• $A$: Provide a plausible factual answer.
• $B$: State that the information is unknown.

Its learned distribution is:

If the training and post-training processes favour direct answers in contexts resembling $X$, the model may assign greater probability to $A$, even when the evidence is insufficient.

This is not the same as consciously deciding to deceive.

It is a consequence of the model's learned response distribution and its limitations in distinguishing supported from unsupported claims.

Importantly, hallucinations can also arise from other causes, including faulty reasoning, misinterpreted evidence, outdated knowledge, and errors when combining information from multiple sources.

Confidence is not the same as correctness

Suppose a model assigns 90% probability to a particular next token.

That does not mean the factual claim being constructed has a 90% chance of being true.

Token probabilities are probabilities over textual continuations under the model.

They are not automatically calibrated probabilities over real-world propositions.

This is a fundamental distinction.

For example:

does not imply:

The second quantity requires a model of the claim, the evidence, and the relationship between them.

A standard language model does not necessarily calculate that quantity explicitly.

7.9 Why more training doesn't eliminate hallucinations

It might seem that sufficiently extensive training would solve the problem.

But consider the distinction between two questions.

Question 1: What is the boiling point of pure water at standard atmospheric pressure?

The answer is well established and widely represented in training material.

Question 2: What did a particular engineer decide in a private meeting yesterday?

The second question cannot be answered reliably from general pretraining alone.

The required information may never have appeared in the training data.

No amount of training on unrelated text can establish the contents of that meeting.

This illustrates an information-theoretic limitation.

If the necessary evidence is unavailable, the model cannot reliably infer an arbitrary hidden fact.

It may make a plausible guess, but plausibility is not knowledge.

The appropriate behaviour is to acknowledge the missing information or obtain additional evidence.

7.10 Retrieval and verification change the problem

One way to reduce hallucinations is to give the model access to relevant evidence.

For example, suppose the user asks:

What does our current safety policy require before offshore maintenance work can begin?

A model relying only on pretraining may know general offshore safety principles.

But it cannot establish what a particular organisation's current policy requires unless that policy is supplied or retrieved.

A more reliable architecture is:

This approach is commonly called retrieval-augmented generation, or RAG.

It changes the task from relying entirely on learned associations to generating an answer using supplied evidence.

However, retrieval alone does not guarantee correctness.

The model may misinterpret a document, overlook a qualification, combine incompatible sources, or cite material that doesn't support its claim.

A trustworthy system therefore needs more than document retrieval. It needs mechanisms for checking source relevance, applicability, consistency, and whether each substantive claim is actually supported.

For consequential decisions, deterministic checks, domain-specific rules, independent validation, and human oversight may also be necessary.

Interactive lab

Separate plausibility from evidence

A fictional policy excerpt says: “Before work begins, confirm permit approval and isolate the equipment.”

Which claim are you checking?The excerpt requires permit approval before work begins.A supervisor approved yesterday’s permit at 09:00.The excerpt says equipment isolation is unnecessary.Check against the excerpt

This small exercise checks only support in the supplied text. It does not establish whether the fictional policy is current, authoritative, or applicable to real work.

7.11 Can an LLM genuinely understand?

We now reach a philosophical and scientific question.

What would count as genuine understanding?

There is no universally agreed definition.

But we can distinguish several possible criteria.

Criterion 1: Behavioural competence

Can the system answer questions, solve problems, and apply knowledge to new situations?

Modern LLMs clearly demonstrate substantial competence in many domains.

They can perform tasks that require sophisticated combinations of language, mathematics, and learned relationships.

This is an empirical observation.

Criterion 2: Internal representations

Does the system contain internal structures that encode meaningful relationships?

Research into neural-network representations provides evidence that models learn structured information about syntax, semantics, entities, and various features of the world.

However, these representations are not always reliable, complete, or straightforwardly interpretable.

Criterion 3: Causal understanding

Can the system distinguish correlation from causation and reason correctly about interventions?

Consider:

Ice cream sales and drowning incidents both increase in summer.

A model may correctly explain that warmer weather influences both variables.

But producing the explanation does not establish that it has a consistently reliable causal model.

A stronger test would involve novel interventions, counterfactual questions, and controlled problems that distinguish causal reasoning from learned correlations.

Criterion 4: Grounded understanding

Does the system connect its representations to observations and actions in the world?

A text-only LLM learns from descriptions of experience rather than direct physical experience.

Multimodal models and systems connected to sensors or tools can have richer forms of grounding.

But grounding does not automatically guarantee accurate understanding.

Criterion 5: Subjective experience

Does the system have conscious experience or subjective awareness?

This is a separate question from whether it can reason or solve problems.

There is currently no generally accepted scientific test that settles whether an AI system has subjective experience.

Consequently, we should not infer consciousness merely from fluent language or apparent reasoning.

Nor should we claim that the mathematics of neural networks alone resolves the philosophical question.

7.12 A useful distinction: competence, reliability, and justification

For practical purposes, I think it helps to separate three questions.

Can the model produce a useful answer?

This is a question of capability.

Will it produce a correct answer consistently?

This is a question of reliability.

Can we establish why the answer should be trusted?

This is a question of justification. We need relevant evidence, valid calculations, and checks appropriate to the claim. Capability, reliability, and justification are related but distinct: one successful answer establishes neither consistent performance nor adequate support for every assertion.

A practical evaluation should test unfamiliar cases, identify failure conditions, check numerical answers independently, and verify the evidence for factual claims. For an arbitrary hidden fact with no available evidence, the right response can be to acknowledge that it is unknown.

Check your understanding

Lesson 7 checkpoint

A model assigns 90% probability to a next token. What can we conclude?

A. The claim being written has a 90% chance of being trueB. The model gives that textual continuation high probability under its learned distributionC. The answer has been independently checkedD. The model has subjective experience

Check answer

The central insight from Lesson 7

Next-token prediction can reward useful representations and reusable computations. Learned competence can coexist with memorisation, brittle generalisation, and hallucination. Fluent language and high token probability alone do not establish truth, reliable causal understanding, or subjective experience.

We have followed the complete path: text becomes tokens; tokens become vectors; learned transformations and attention produce contextual representations; training adjusts the parameters; decoding generates a continuation. Evidence and verification determine how much confidence a particular answer deserves.