Attention on SciDraft | A Hugo Theme

Causal Multi-Head Attention in Python (intermediate)

Tue, 22 Sep 2026 07:10:00 +0900

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Attention is the one place in a transformer where tokens exchange information: every other block — the projections, the activations, the normalisations — treats positions independently. That makes attention mostly shape bookkeeping, and the shapes alone decide which products are legal.

Attention with Two Tokens (example)

Tue, 22 Sep 2026 07:05:00 +0900

The smallest interesting attention example has two positions and one head, which is enough for a full calculation with no matrix product larger than two by two.

Working it at this size makes the two roles visible: the score decides how much the output at position two borrows from position one, and the value vectors set what is borrowed. With a causal mask, the first row has nothing to attend to but itself.

Attention Mechanism: Pseudocode

Mon, 21 Sep 2026 15:32:00 +0900

Attention lets a model decide, for each position it is processing, which other positions to draw information from. This note states the scaled dot-product form as an algorithm rather than as an equation, so the sequence of operations — and the shape of each intermediate result — is explicit.

Masked Softmax Attention (algorithm)

Mon, 21 Sep 2026 14:52:00 +0900

An algorithm is a procedure with a specification, and this one says how the attention definition is executed on a machine: scale the scores, remove the positions that are not allowed, normalise and mix.

The KV Cache Is Exact (claim)

Mon, 21 Sep 2026 14:22:00 +0900

Caching keys and values is standard practice, and the claim that justifies it is strong: incremental decoding gives exactly the same outputs as recomputing attention over the whole sequence, not an approximation of them.

Attention Variants by Cost and Cache (quadrant chart)

Mon, 21 Sep 2026 14:21:00 +0900

Choosing an attention variant is a two-constraint problem rather than a ranking: the score cost decides how long a sequence can be trained on, and the per-token cache decides how long a sequence can be served, and the two orderings of the variants are not the same.

Attention Entropy Falls with Depth (observation)

Mon, 21 Sep 2026 11:02:00 +0900

Entropy is a convenient scalar summary of an attention row, and the observation reported here is that its average falls as one moves up the layers: early heads spread their weight widely, later heads concentrate it.

Attention Outputs Are Convex Combinations (proposition)

Mon, 21 Sep 2026 10:22:00 +0900

Attention is often drawn as a lookup, and the proposition below says what kind of lookup it is: a convex combination of the value vectors, with weights that are non-negative and sum to one because they came out of a softmax.

Proof: Attention Outputs Lie in the Convex Hull (proof)

Mon, 21 Sep 2026 09:52:00 +0900

The statement that attention outputs are convex combinations follows from two facts about the softmax and one line of the triangle inequality, and the proof shows exactly where each is used.

Families of Attention (mindmap)

Mon, 21 Sep 2026 09:44:00 +0900

The variants of attention that appear in long-context work are easier to keep apart by what they replace than by their names: each one gives up the full score matrix and puts something else in its place, and the something else is what the family is called.

Scaled Dot-Product Attention (definition)

Mon, 21 Sep 2026 09:12:00 +0900

Definitions in this subject fix notation rather than prove anything, and attention is the clearest case: three projections, one score matrix, one weighted average. Every implementation, diagram and proof in the rest of the literature is written against this one paragraph of notation.

Attention Variants for Long Contexts (striped table)

Mon, 21 Sep 2026 07:32:00 +0900

Long-context attention is usually chosen on cost rather than on quality. What decides whether a variant is usable at a given sequence length is how the score cost and the per-token cache grow with \(T\), and both are easiest to compare side by side in one table rather than across several paragraphs.

Notation for Multi-Head Attention (academic table)

Mon, 21 Sep 2026 07:12:00 +0900

Multi-head attention reads better as shape bookkeeping than as prose: every symbol below is a tensor, and the shapes alone decide which products are legal, which is why papers about attention variants usually open with a notation table before showing a single result.