Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. Once a model trains, the folder structure has a second job: it has to separate what changes per run from what changes per idea. Configurations and checkpoints belong to a run; attention code, data preparation and the training loop belong to the project, and the split is what lets a result be reproduced from a config file plus a commit rather than from memory. ...
Demo
Proof: Attention Outputs Lie in the Convex Hull (proof)
Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. The statement that attention outputs are convex combinations follows from two facts about the softmax and one line of the triangle inequality, and the proof shows exactly where each is used. ...
Families of Attention (mindmap)
Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. The variants of attention that appear in long-context work are easier to keep apart by what they replace than by their names: each one gives up the full score matrix and puts something else in its place, and the something else is what the family is called. ...
Sylvester's Criterion for Positive Definiteness (criterion)
Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. Positive definiteness is the condition that decides whether a quadratic form curves upwards, and Sylvester’s criterion turns it into a finite computation: look at the determinants of the leading blocks of the matrix, from the top left outwards. ...
The Gradient of Cross-Entropy Through Softmax (proposition)
Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. A proposition is a proved statement of moderate weight; this one is the reason the training loss needs no special treatment in the backward pass. Composing softmax with negative log-likelihood, the Jacobian of the softmax and the derivative of the logarithm cancel almost completely. ...
Maximum Entropy Gives the Softmax (theorem)
Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. The exponential form of a language model’s output is not a design choice made for convenience; it is forced by an optimisation over distributions. Maximise entropy subject to matching a set of expected features, and the solution is always of the form \(\exp(\theta^{\top} f(x))\), normalised. ...
Scaled Dot-Product Attention (definition)
Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. Definitions in this subject fix notation rather than prove anything, and attention is the clearest case: three projections, one score matrix, one weighted average. Every implementation, diagram and proof in the rest of the literature is written against this one paragraph of notation. ...
Cross-Entropy (definition)
Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. The training objective of a language model has a name and a formula, and both matter when the loss curve is read: cross-entropy is the expected negative log-probability of what actually happened, measured in nats. ...