Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Once a model trains, the folder structure has a second job: it has to separate what changes per run from what changes per idea. Configurations and checkpoints belong to a run; attention code, data preparation and the training loop belong to the project, and the split is what lets a result be reproduced from a config file plus a commit rather than from memory.

This tree is three levels deep and gives each responsibility its own folder: model code, data preparation next to its shards, the entry points under scripts/, and gradient checks under tests/. The two generated folders are listed without their contents, since their files are produced by running the project rather than written into it.

tiny-transformer/
├── configs/
│   ├── base.toml           # 6 layers, 8 heads, 2k context
│   └── smoke.toml          # 2 layers, for a one-minute run
├── data/
│   ├── prepare.py          # tokenise the corpus and shard it
│   └── shards/
│       ├── train.bin
│       └── val.bin
├── model/
│   ├── attention.py        # scaled dot-product, causal mask
│   ├── blocks.py           # residual stream, layer norm, MLP
│   └── loss.py             # softmax cross-entropy
├── scripts/
│   ├── train.py            # reads a config, writes a checkpoint
│   ├── eval.py             # validation loss for one checkpoint
│   └── sample.py           # top-p decoding from a checkpoint
├── checkpoints/            # written by scripts/train.py
├── tests/
│   └── test_gradients.py   # finite-difference checks per module
└── README.md

The layout pays off at evaluation time, which is where most projects lose their history: a folder of checkpoints plus one evaluation script regenerates the same number months later, and the config beside each checkpoint says which model produced it. The next size up is what appears when a project holds several experiments at once — sweeps, benchmarks and figures — and needs each of them to stay traceable.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

The statement that attention outputs are convex combinations follows from two facts about the softmax and one line of the triangle inequality, and the proof shows exactly where each is used.

Writing it out also shows what would break it. Remove the normalisation of the weights, or allow them to be negative, and the argument fails at the first step — the outputs are then free to leave the hull, which is what happens in attention variants that drop the softmax.

Proof. Let \(A\) be the row-stochastic matrix of attention weights, so \(A_{ij} \geq 0\) and \(\sum_{j} A_{ij} = 1\) for every \(i\), and let \(o_i = \sum_j A_{ij} v_j\).

Step one, the weights form a distribution: both properties follow from the definition of the softmax together with the fact that exponentials are strictly positive.

Step two, the output is a convex combination: a sum of the form \(\sum_j A_{ij} v_j\) with non-negative coefficients adding to one is the definition of a convex combination, so \(o_i\) lies in the convex hull of the value vectors by definition of the hull.

Step three, the norm bound: by the triangle inequality and \(\sum_j A_{ij} = 1\),

Equality in the last step requires a single non-zero weight on a maximal vector, which is the limit of a fully peaked attention row.

The proof is short because the softmax does the work up front: any normalisation that produces non-negative weights summing to one gives the same conclusion, which is why the hull argument is quoted for attention without inspecting the score function at all.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

The variants of attention that appear in long-context work are easier to keep apart by what they replace than by their names: each one gives up the full score matrix and puts something else in its place, and the something else is what the family is called.

This note uses a mind map, which suits a taxonomy with no order to it: the four families below are siblings, and the leaves under each are the mechanism rather than a sequence of steps.

mindmap
  root((Attention))
    Full[Full attention]
      Dense[scores over every earlier position]
      Cost[cost grows with T squared]
    Sparse[Sparse patterns]
      Window[sliding window of the last w positions]
      Blocks[fixed pattern of blocks]
    Linear[Linear attention]
      Kernel[feature map replaces the softmax]
      Recurrent[running summary instead of a cache]
    State[State space]
      Fixed[state of fixed size]
      Decay[decay stands in for position]

Read this way, the families differ in what they keep: a sparse pattern keeps the softmax and drops most of its entries, a linear method keeps all the entries and drops the softmax, and a state-space model drops the pairwise structure altogether in favour of a recurrence.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Positive definiteness is the condition that decides whether a quadratic form curves upwards, and Sylvester’s criterion turns it into a finite computation: look at the determinants of the leading blocks of the matrix, from the top left outwards.

The criterion is used wherever a second-order quantity has to be checked — a Hessian for a local minimum, a covariance for a Gaussian density, a preconditioner for a second-order optimiser — and each of those settings inherits the same test.

Criterion (Sylvester). A symmetric matrix \(M \in \mathbb{R}^{n \times n}\) is positive definite if and only if every leading principal minor is positive:

Two checks worth keeping alongside it: a positive definite matrix has all eigenvalues positive and hence a non-zero determinant, so the criterion is consistent with the spectral definition; and replacing \(\gt\) by \(\geq\) gives the test for positive semi-definiteness, where the minors need not all be positive, which is why the two cases are stated separately.

For a machine-learning problem the criterion is mostly used to diagnose failure: a Hessian whose minors change sign marks a saddle rather than a minimum, which is what makes the distinction visible in a computed matrix even when the loss curve is flat.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

A proposition is a proved statement of moderate weight; this one is the reason the training loss needs no special treatment in the backward pass. Composing softmax with negative log-likelihood, the Jacobian of the softmax and the derivative of the logarithm cancel almost completely.

What survives is an expression that can be read in one line: the gradient with respect to the scores is the predicted distribution minus the observed one.

Proposition. Let \(p = \operatorname{softmax}(z)\) for a score vector \(z \in \mathbb{R}^{V}\), and let the loss be the negative log-likelihood of a target token \(y\), so that \(L = -\log p_y\). Then

where \(e_y\) is the one-hot vector of the target. For a batch of \(N\) positions, the mean loss has gradient \((p - e_y) / N\) at each of them.

The expression is what makes the backward pass through a language-model head cheap, since the gradient at every position is already available as the forward pass output, and it is the quantity a finite-difference check is normally used to verify.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

The exponential form of a language model’s output is not a design choice made for convenience; it is forced by an optimisation over distributions. Maximise entropy subject to matching a set of expected features, and the solution is always of the form \(\exp(\theta^{\top} f(x))\), normalised.

For a token distribution constrained only by the scores, that form is the softmax. The theorem is the reason the softmax keeps reappearing in places that look unrelated to classification.

Theorem (maximum entropy). Among distributions \(q\) on a finite set that satisfy \(\mathbb{E}_q[f] = \bar{f}\) for a given feature map \(f\), the one maximising the entropy \(H(q) = -\sum_v q_v \log q_v\) has the form

for some parameter vector \(\theta\) chosen so the constraints hold. With \(f(v) = z_v\) and \(\theta = 1\), this is exactly the softmax of the scores.

Read backwards, the theorem says what a softmax commits to: it is the smoothest distribution consistent with the scores, so any structure beyond them — sparsity, hard constraints — has to be imposed deliberately rather than falling out of the parameterisation.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Definitions in this subject fix notation rather than prove anything, and attention is the clearest case: three projections, one score matrix, one weighted average. Every implementation, diagram and proof in the rest of the literature is written against this one paragraph of notation.

The definition below is stated for a single sequence without the batch dimension. The softmax is taken over the keys, so each row of the score matrix becomes a distribution over the positions a token may read from.

Definition (scaled dot-product attention). Let \(Q, K, V\) be matrices of shape \(T \times d_h\), called queries, keys and values. The attention output is

where the softmax is applied to every row of the \(T \times T\) score matrix \(QK^{\top} / \sqrt{d_h}\), and \(\sqrt{d_h}\) is a fixed rescaling that keeps the scores of comparable size as the head width grows.

Two details in the definition carry most of the consequences: the softmax is over the keys rather than the values, which makes the output a weighted average of value vectors, and the \(\sqrt{d_h}\) factor makes the score variance independent of the head width under the usual initialisation.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

The training objective of a language model has a name and a formula, and both matter when the loss curve is read: cross-entropy is the expected negative log-probability of what actually happened, measured in nats.

The definition is written here for a single position, then specialised to the case a language model is always in — a target distribution concentrated on one token.

Definition (cross-entropy). For two probability distributions \(p\) and \(q\) on the same finite set, the cross-entropy of \(q\) relative to \(p\) is

with the convention \(0 \log 0 = 0\). When \(p\) is the one-hot distribution of the token that actually came next, the sum collapses to a single term,

which is the quantity minimised at every position of a training sequence.

Writing the one-hot case separately is what makes the loss comparable across vocabularies of different sizes: the generic form says what the objective is, and the collapsed form says what is actually computed, per token, per position.