Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

A corollary is what falls out of a theorem with one substitution, and the useful one here is a floor on perplexity. Since cross-entropy cannot fall below the entropy of the data, and entropy is non-negative, perplexity cannot fall below one.

The bound is what makes a reported perplexity of 0.9 a bug report rather than a result: no distribution can assign more probability than certainty to every observed token.

Corollary. For any model distribution \(q\) and any data distribution \(p\) with finite support,

with equality throughout if and only if \(p\) is concentrated on a single outcome and \(q = p\) there.

The first inequality is Gibbs’ inequality composed with the exponential, which is increasing; the second is \(H(p) \geq 0\), which holds because each term \(-p_v \log p_v\) is non-negative.

In practice the floor is far above one: natural text has entropy in the low nats per token, so the gap between a model’s perplexity and one is not the gap worth tracking — the gap to the entropy of the data is.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

A principle is a guide to design rather than a result about a particular model, and this one is the oldest in statistics: expected error splits into a term from the model being too simple, a term from it being too sensitive to the sample, and a term that no model can remove.

Read as a design principle, it says that capacity and regularisation are two ends of one dial, and that a change which reduces one term usually pays in the other. That reading is what makes early stopping, weight decay and dataset size comparable interventions.

Principle (bias-variance decomposition). For an estimator \(\hat{f}\) of a target \(f\) at an input \(x\), with expectation taken over training samples,

The principle is the reading of the identity: increasing capacity tends to lower the bias term and raise the variance term, and the noise term is fixed by the data.

In a large model the two terms are usually discussed under different names — underfitting and overfitting — and the practical consequence of the decomposition is that neither can be diagnosed from the training loss alone, since the variance term is only visible on data the model has not seen.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

A formula is a rule written in symbols so that it can be evaluated, and the one nearly every model is trained with is Adam’s update: two running averages of the gradient, a correction for the fact that both start at zero, and a step whose size is set by their ratio.

The bias-correction step is what the formula is usually remembered for. Without it the first few steps are far too large, because a running average that began at zero underestimates the quantity it is meant to summarise.

Formula (Adam). With parameters \(\theta\), gradient \(g_t\) at step \(t\), step size \(\eta\), decay rates \(\beta_1, \beta_2 \in [0, 1)\) and a small constant \(\varepsilon\),

The squared gradient in \(v_t\) is taken element-wise, so the division is also element-wise: each parameter receives a step inversely proportional to the recent size of its own gradient. The defaults \(\beta_1 = 0.9\), \(\beta_2 = 0.999\), \(\varepsilon = 10^{-8}\) come from the original paper and are still the common setting.

The division by \(\sqrt{\hat{v}_t}\) is what makes the method scale-free: multiplying the loss by a constant leaves the update nearly unchanged, which is why the same step size works across architectures that produce gradients of very different magnitudes.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Entropy is a convenient scalar summary of an attention row, and the observation reported here is that its average falls as one moves up the layers: early heads spread their weight widely, later heads concentrate it.

Like most observations about trained networks, this is a tendency rather than a rule, and the note states the measured form and the conditions under which it was measured. The useful part is the consequence for pruning and for interpretability, both of which care whether a head is diffuse or focused.

Observation. For a trained decoder with \(L\) layers and \(H\) heads per layer, let \(A^{(\ell, h)}\) be the attention matrix at layer \(\ell\) and head \(h\), averaged over a held-out set of sequences. With the row entropy

the observation is a decreasing trend in \(\mathcal{H}^{(\ell, h)}\) as \(\ell\) increases, from values close to \(\log T\) in the first layers towards values well below it in the last ones. Individual heads depart from the trend in both directions.

Average entropy is not the same as usefulness: a head that attends uniformly has maximum entropy and carries little information, while a head that always attends to the same relative offset can have low entropy and be essential, so the observation is a description of the population rather than a criterion for any single head.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

A lemma is a small result proved on the way to something larger, and this one is proved in two lines yet carries most of the numerical stability of a softmax: adding the same constant to every score changes nothing.

The consequence is the implementation that is actually used — subtract the largest score before exponentiating — and the reason no probability in the output can overflow to infinity while another underflows to zero.

Lemma (shift invariance). For any vector \(z \in \mathbb{R}^{n}\) and any constant \(c\),

where \(\mathbf{1}\) is the all-ones vector.

With \(c = -\max_i z_i\), the largest exponent becomes \(e^{0} = 1\), so every term of the sum lies in \((0, 1]\) and the computation stays in range for scores of any magnitude.

The lemma is a statement about the exact function; floating-point arithmetic only approximates it, which is why the shifted form is preferred in code — it is the version whose rounding errors stay small for the range of scores a model actually produces.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

The simplest optimiser with a memory is gradient descent with momentum: keep a running velocity, add the gradient to it, and step in the direction of the velocity rather than the gradient. The algorithm is short, and the behaviour it introduces is worth the two extra lines.

The effect is an averaging of successive gradients, which cancels the oscillating part of the noise and amplifies the consistent part. Its cost is a second state vector and one more constant to set.

Algorithm (heavy-ball momentum). Given a differentiable loss \(L\), a step size \(\eta \gt 0\), a momentum coefficient \(\beta \in [0, 1)\), an initial parameter vector \(\theta_0\) and zero initial velocity \(v_0\), the steps are:

1. Sample or compute the gradient \(g_t = \nabla L(\theta_{t-1})\).
2. Update the velocity: \(v_t = \beta v_{t-1} + g_t\).
3. Update the parameters: \(\theta_t = \theta_{t-1} - \eta v_t\).

Unrolling the recursion shows that the update is a weighted sum of past gradients with geometrically decaying weights,

so the effective number of gradients being averaged is \(1 / (1 - \beta)\): \(\beta = 0.9\) averages roughly the last ten steps.

The recursion also explains the pathology of a badly chosen coefficient: with \(\beta\) close to one the effective averaging window is long enough for the direction to lag behind a changing loss surface, and the momentum carries the parameters past the minimum it was approaching.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Attention is often drawn as a lookup, and the proposition below says what kind of lookup it is: a convex combination of the value vectors, with weights that are non-negative and sum to one because they came out of a softmax.

The statement constrains the output geometry before any training happens, and it is the reason an attention output cannot be extrapolated outside the region its value vectors span — no matter how extreme the queries become.

Proposition. Let \(A \in \mathbb{R}^{T \times T}\) be the row-stochastic matrix produced by the softmax in attention, so that \(A_{ij} \geq 0\) and \(\sum_j A_{ij} = 1\) for every row \(i\). Then each output row \(o_i = \sum_j A_{ij} v_j\) lies in the convex hull of the value vectors \(v_1, \dots, v_T\).

A one-line consequence is a bound on the output,

by the triangle inequality and \(\sum_j A_{ij} = 1\).

The bound is inherited rather than learned: making attention more confident moves the output towards a vertex of the hull, and no amount of training can move it outside.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

The same generation loop can be described without messages, as a machine with a small number of states. This view is useful when the interesting questions are about transitions — when decoding starts, what keeps it running, and which conditions end it. The state diagram below is the whole lifecycle of one completion.

Two conditions drive the transitions. The first is the probability the model assigns to the end-of-sequence token: decoding stops when it becomes the sampled token, \(x_t = \langle \text{eos} \rangle\). The second is a hard budget — a maximum number of tokens — which stops generation that has not converged, and is the reason a truncated answer can look unfinished.

stateDiagram-v2
    [*] --> Idle
    Idle --> Prefill: prompt received
    Prefill --> Decode: caches built
    Decode --> Decode: next token
    Decode --> Complete: EOS sampled
    Decode --> Truncated: budget spent
    Complete --> [*]
    Truncated --> [*]

Reading the diagram. Prefill and Decode are separate states because they cost differently: prefill processes the prompt in parallel, decode advances one position at a time and is bound by memory traffic rather than arithmetic. The self-loop on Decode is where almost all the time is spent. Note also that the two exit states are not equivalent — Complete means the model chose to stop, Truncated means something else did, and downstream code that treats them as the same will occasionally present a cut-off answer as a finished one.