Mermaid on SciDraft | A Hugo Theme

Autoregressive Generation (sequence diagram)

Tue, 22 Sep 2026 07:15:00 +0900

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

A language model writes one token at a time. The prompt is processed once, then each new token is appended to the sequence and fed back in, so generation is a loop between the model and the sampler rather than a single call. The sequence diagram below shows the participants and the messages between them.

Attention Variants by Cost and Cache (quadrant chart)

Mon, 21 Sep 2026 14:21:00 +0900

Choosing an attention variant is a two-constraint problem rather than a ranking: the score cost decides how long a sequence can be trained on, and the per-token cache decides how long a sequence can be served, and the two orderings of the variants are not the same.

A Transformer Training Step (flowchart)

Mon, 21 Sep 2026 14:02:00 +0900

One training step is a fixed cycle: the model predicts, the predictions are scored against the targets, the error is propagated backwards, and the parameters move a little in the direction that reduces it. The diagram below shows that cycle; the repetition is the whole of training.

A Training Schedule (gantt)

Mon, 21 Sep 2026 13:00:00 +0900

A training run has a shape in time as well as in the loss curve: data preparation that nothing else can start without, a warmup whose length is set by the step size, a long stable phase, and a cooldown whose purpose is to end at a point the optimiser has settled into.

Decoding as a State Machine (state diagram)

Mon, 21 Sep 2026 10:12:00 +0900

The same generation loop can be described without messages, as a machine with a small number of states. This view is useful when the interesting questions are about transitions — when decoding starts, what keeps it running, and which conditions end it. The state diagram below is the whole lifecycle of one completion.

Families of Attention (mindmap)

Mon, 21 Sep 2026 09:44:00 +0900

The variants of attention that appear in long-context work are easier to keep apart by what they replace than by their names: each one gives up the full score matrix and puts something else in its place, and the something else is what the family is called.