SciDraft | A Hugo Theme

A Paper with Multiple References (demo)

Tue, 22 Sep 2026 07:20:00 +0900

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

A long paper sometimes needs more than one bibliography: a main text and an appendix written against a different literature, or two halves of a review whose sources barely overlap. The theme supports that by letting each list name its own file, so each list downloads only its own entries and numbers them from one. This note is written that way on purpose — the first half is about how a model is asked, the second about how it is adapted, and the two halves share no sources.

Autoregressive Generation (sequence diagram)

Tue, 22 Sep 2026 07:15:00 +0900

A language model writes one token at a time. The prompt is processed once, then each new token is appended to the sequence and fed back in, so generation is a loop between the model and the sampler rather than a single call. The sequence diagram below shows the participants and the messages between them.

Causal Multi-Head Attention in Python (intermediate)

Tue, 22 Sep 2026 07:10:00 +0900

Attention is the one place in a transformer where tokens exchange information: every other block — the projections, the activations, the normalisations — treats positions independently. That makes attention mostly shape bookkeeping, and the shapes alone decide which products are legal.

Attention with Two Tokens (example)

Tue, 22 Sep 2026 07:05:00 +0900

The smallest interesting attention example has two positions and one head, which is enough for a full calculation with no matrix product larger than two by two.

Working it at this size makes the two roles visible: the score decides how much the output at position two borrows from position one, and the value vectors set what is borrowed. With a causal mask, the first row has nothing to attend to but itself.

Layout of a Scaling Study (advanced folder structure)

Tue, 22 Sep 2026 07:01:00 +0900

A project that produces a paper has to keep three kinds of thing apart that are easy to mix: the installed source, the outputs of individual runs, and the material the paper is written from. Once several sweeps and a performance benchmark live in the same repository, the folder structure is what decides whether a figure can be traced back to the run that produced it.

A Survey of Chain-of-Thought Prompting: Single Chains, Sampling and Demonstration Construction (survey)

Tue, 22 Sep 2026 06:55:00 +0900

Chain-of-thought prompting began as a single sentence appended to a prompt and is now a family of strategies that differ in what they add: a fixed reasoning skeleton, several sampled chains and a vote, demonstrations the method writes for itself, or calls to an external tool in the middle of the reasoning. This survey groups the entries in its bibliography by that question, following the framing of the earlier survey of prompting strategies \cite{Yu2023-qr}, and asks in each group what the reported gain is actually attributed to.

The Variance Identity (identity)

Mon, 21 Sep 2026 16:22:00 +0900

An identity is an equation that holds for every admissible value of its variables, and this one converts a variance into two expectations, which is often the difference between a quantity that can be estimated in one pass and one that cannot.

Bayes' Rule (rule)

Mon, 21 Sep 2026 16:12:00 +0900

Bayes’ rule is the rule for turning a model’s likelihood into a statement about hypotheses, and it is used twice in a language-model workflow: once to reason about uncertainty in evaluation, and once whenever a prior over prompts or parameters is combined with observed data.

The Chain Rule (rule)

Mon, 21 Sep 2026 16:02:00 +0900

A rule is a procedure applied so often that it stops looking like a theorem, and the chain rule is the extreme case: every backward pass through a stacked model is one long application of it.

Layout of a Small Numeric Experiment (starter folder structure)

Mon, 21 Sep 2026 15:52:00 +0900

The smallest useful project around a piece of neural-network mathematics holds one idea per file and keeps the generated numbers out of the code. Nothing in it is conventional yet: a starter folder is decided by what you want to compare next, not by what a template says.

Backpropagation Through a Dense Layer

Mon, 21 Sep 2026 15:42:00 +0900

Backpropagation computes, for one layer, three gradients: one for each parameter of the layer and one to hand to the layer before it. The third is what makes deep networks trainable — without it, error could not travel past the last layer. The procedure below is the backward pass of a single dense layer.

Attention Mechanism: Pseudocode

Mon, 21 Sep 2026 15:32:00 +0900

Attention lets a model decide, for each position it is processing, which other positions to draw information from. This note states the scaled dot-product form as an algorithm rather than as an equation, so the sequence of operations — and the shape of each intermediate result — is explicit.

Inner-Product Axioms (axiom)

Mon, 21 Sep 2026 15:22:00 +0900

Attention is built on a quantity that looks like a similarity but is defined by three axioms: an inner product is symmetric, linear in each argument, and positive definite. Those three properties are what allow a score to be read as “how much one vector points along another”.

Rotary Position Embedding (formula)

Mon, 21 Sep 2026 15:12:00 +0900

Position information has to enter attention somewhere, and one way is to rotate the query and key vectors by a position-dependent angle. The formula is appealing because the effect on the score depends only on the difference of the two positions.

Validation Loss by Model Size (bordered table)

Mon, 21 Sep 2026 15:02:00 +0900

A scaling study compares models of different sizes trained on the same token budget. If the loss falls smoothly as capacity grows, the family is in a regime where parameters still buy accuracy, and the fall can be summarised as a power law in the parameter count rather than reported as a list of separate runs.

Masked Softmax Attention (algorithm)

Mon, 21 Sep 2026 14:52:00 +0900

An algorithm is a procedure with a specification, and this one says how the attention definition is executed on a machine: scale the scores, remove the positions that are not allowed, normalise and mix.

The Woodbury Identity (identity)

Mon, 21 Sep 2026 14:42:00 +0900

The Woodbury identity rewrites the inverse of a matrix plus a low-rank correction in terms of the inverse of the original matrix and a much smaller inverse. It is the algebraic fact behind every method that updates a large system cheaply.

Features Are Directions (conjecture)

Mon, 21 Sep 2026 14:32:00 +0900

The linear representation hypothesis conjectures that the concepts a network has learned are encoded as directions in activation space, so that adding a concept vector to an activation moves the computation in a meaningful way.

The KV Cache Is Exact (claim)

Mon, 21 Sep 2026 14:22:00 +0900

Caching keys and values is standard practice, and the claim that justifies it is strong: incremental decoding gives exactly the same outputs as recomputing attention over the whole sequence, not an approximation of them.

Attention Variants by Cost and Cache (quadrant chart)

Mon, 21 Sep 2026 14:21:00 +0900

Choosing an attention variant is a two-constraint problem rather than a ranking: the score cost decides how long a sequence can be trained on, and the per-token cache decides how long a sequence can be served, and the two orderings of the variants are not the same.

A Transformer Training Step (flowchart)

Mon, 21 Sep 2026 14:02:00 +0900

One training step is a fixed cycle: the model predicts, the predictions are scored against the targets, the error is propagated backwards, and the parameters move a little in the direction that reduces it. The diagram below shows that cycle; the repetition is the whole of training.

Kolmogorov Axioms for a Token Distribution (axiom)

Mon, 21 Sep 2026 13:32:00 +0900

An axiom is an assumption taken as given, and the ones that hold up a language model are Kolmogorov’s three: probabilities are non-negative, the whole space has probability one, and disjoint events add. Everything else about token distributions is derived from them rather than assumed.

Softmax of Three Scores (example)

Mon, 21 Sep 2026 13:12:00 +0900

An example is where a formula stops being notation, and this one is small enough to check by hand: three scores, one softmax, three probabilities that must sum to one.

The numbers below are the ones the companion snippet prints, so the arithmetic here and the output of a few lines of code can be compared directly.

A Bound on the Attention Output Norm (corollary)

Mon, 21 Sep 2026 13:02:00 +0900

Reading a bound off a proposition is the standard use of a corollary, and the one worth having here is a ceiling on how large an attention output can be. Since the output is a convex combination of value vectors, its length can never exceed the longest of them.

A Training Schedule (gantt)

Mon, 21 Sep 2026 13:00:00 +0900

A training run has a shape in time as well as in the loss curve: data preparation that nothing else can start without, a warmup whose length is set by the step size, a long stable phase, and a cooldown whose purpose is to end at a point the optimiser has settled into.

Softmax Cross-Entropy and Its Gradient in Python (advanced)

Mon, 21 Sep 2026 12:52:00 +0900

The number a language model reports while training is the mean negative log-likelihood of the token that actually came next: softmax cross-entropy over the vocabulary at every position. Its backward pass has an unusually clean closed form — the softmax Jacobian and the logarithm cancel, leaving the probabilities minus the one-hot target, divided by the number of predictions.

Smoothness of the Loss (postulate)

Mon, 21 Sep 2026 12:32:00 +0900

Step sizes in deep learning are chosen as if the loss were smooth in its parameters, and that is an assumption, not a theorem. It is the smoothness postulate: the gradient does not change arbitrarily fast, so a small step in parameter space produces a predictable change in loss.

Proof of the Softmax Shift Lemma (proof)

Mon, 21 Sep 2026 12:22:00 +0900

A proof is the part of a note that cannot be skipped, and this one is short enough to show the whole pattern: write the definition, factor out the constant, use that it cancels between numerator and denominator.

Matrix Products Do Not Commute (counterexample)

Mon, 21 Sep 2026 12:12:00 +0900

A counterexample is the cheapest way to kill a plausible statement, and the one worth having ready concerns the order of matrix multiplication. The tempting assertion that \(AB = BA\) fails for almost every pair of matrices, and it fails on the smallest possible example.

Log-Sum-Exp Bounds (lemma)

Mon, 21 Sep 2026 12:02:00 +0900

The log-partition function appears wherever a normalising constant is needed, and it is trapped between two elementary bounds that are enough for most arguments: the largest term below, the largest term plus the log of the number of terms above.

Equivariance as a Design Principle (principle)

Mon, 21 Sep 2026 11:52:00 +0900

Architectures are usually chosen for what they can compute, but the sharper criterion is what they cannot distinguish: matching the symmetries of the problem removes the need to learn them from data. This principle is why convolution appears in vision and why attention appears in sequences.

The Polyak-Lojasiewicz Criterion (criterion)

Mon, 21 Sep 2026 11:42:00 +0900

A criterion is a condition that decides whether an argument or a method applies, and this one is the usual substitute for convexity in the analysis of gradient descent: it bounds the loss by the size of its gradient, so a small gradient implies a small excess loss.

Perplexity Is at Least One (corollary)

Mon, 21 Sep 2026 11:32:00 +0900

A corollary is what falls out of a theorem with one substitution, and the useful one here is a floor on perplexity. Since cross-entropy cannot fall below the entropy of the data, and entropy is non-negative, perplexity cannot fall below one.

The Bias-Variance Trade-Off (principle)

Mon, 21 Sep 2026 11:22:00 +0900

A principle is a guide to design rather than a result about a particular model, and this one is the oldest in statistics: expected error splits into a term from the model being too simple, a term from it being too sensitive to the sample, and a term that no model can remove.

The Adam Update (formula)

Mon, 21 Sep 2026 11:12:00 +0900

A formula is a rule written in symbols so that it can be evaluated, and the one nearly every model is trained with is Adam’s update: two running averages of the gradient, a correction for the fact that both start at zero, and a step whose size is set by their ratio.

Attention Entropy Falls with Depth (observation)

Mon, 21 Sep 2026 11:02:00 +0900

Entropy is a convenient scalar summary of an attention row, and the observation reported here is that its average falls as one moves up the layers: early heads spread their weight widely, later heads concentrate it.

Softmax Is Invariant to a Shift (lemma)

Mon, 21 Sep 2026 10:52:00 +0900

A lemma is a small result proved on the way to something larger, and this one is proved in two lines yet carries most of the numerical stability of a softmax: adding the same constant to every score changes nothing.

Gradient Descent with Momentum (algorithm)

Mon, 21 Sep 2026 10:32:00 +0900

The simplest optimiser with a memory is gradient descent with momentum: keep a running velocity, add the gradient to it, and step in the direction of the velocity rather than the gradient. The algorithm is short, and the behaviour it introduces is worth the two extra lines.

Attention Outputs Are Convex Combinations (proposition)

Mon, 21 Sep 2026 10:22:00 +0900

Attention is often drawn as a lookup, and the proposition below says what kind of lookup it is: a convex combination of the value vectors, with weights that are non-negative and sum to one because they came out of a softmax.

Decoding as a State Machine (state diagram)

Mon, 21 Sep 2026 10:12:00 +0900

The same generation loop can be described without messages, as a machine with a small number of states. This view is useful when the interesting questions are about transitions — when decoding starts, what keeps it running, and which conditions end it. The state diagram below is the whole lifecycle of one completion.

Layout of a Trainable Model (intermediate folder structure)

Mon, 21 Sep 2026 10:02:00 +0900

Once a model trains, the folder structure has a second job: it has to separate what changes per run from what changes per idea. Configurations and checkpoints belong to a run; attention code, data preparation and the training loop belong to the project, and the split is what lets a result be reproduced from a config file plus a commit rather than from memory.

Proof: Attention Outputs Lie in the Convex Hull (proof)

Mon, 21 Sep 2026 09:52:00 +0900

The statement that attention outputs are convex combinations follows from two facts about the softmax and one line of the triangle inequality, and the proof shows exactly where each is used.

Families of Attention (mindmap)

Mon, 21 Sep 2026 09:44:00 +0900

The variants of attention that appear in long-context work are easier to keep apart by what they replace than by their names: each one gives up the full score matrix and puts something else in its place, and the something else is what the family is called.

Sylvester's Criterion for Positive Definiteness (criterion)

Mon, 21 Sep 2026 09:42:00 +0900

Positive definiteness is the condition that decides whether a quadratic form curves upwards, and Sylvester’s criterion turns it into a finite computation: look at the determinants of the leading blocks of the matrix, from the top left outwards.

The Gradient of Cross-Entropy Through Softmax (proposition)

Mon, 21 Sep 2026 09:32:00 +0900

A proposition is a proved statement of moderate weight; this one is the reason the training loss needs no special treatment in the backward pass. Composing softmax with negative log-likelihood, the Jacobian of the softmax and the derivative of the logarithm cancel almost completely.

Maximum Entropy Gives the Softmax (theorem)

Mon, 21 Sep 2026 09:22:00 +0900

The exponential form of a language model’s output is not a design choice made for convenience; it is forced by an optimisation over distributions. Maximise entropy subject to matching a set of expected features, and the solution is always of the form \(\exp(\theta^{\top} f(x))\), normalised.

Scaled Dot-Product Attention (definition)

Mon, 21 Sep 2026 09:12:00 +0900

Definitions in this subject fix notation rather than prove anything, and attention is the clearest case: three projections, one score matrix, one weighted average. Every implementation, diagram and proof in the rest of the literature is written against this one paragraph of notation.

Cross-Entropy (definition)

Mon, 21 Sep 2026 09:02:00 +0900

The training objective of a language model has a name and a formula, and both matter when the loss curve is read: cross-entropy is the expected negative log-probability of what actually happened, measured in nats.

The Gram Matrix Is Positive Semi-Definite (property)

Mon, 21 Sep 2026 08:52:00 +0900

The matrix of pairwise inner products of a set of vectors has a property that constrains what similarity patterns are possible: it is symmetric and positive semi-definite, so its eigenvalues are non-negative and its diagonal entries are the squared lengths of the vectors.

Power Iteration with a Convergence Check (pseudocode)

Mon, 21 Sep 2026 08:39:00 +0900

The largest singular value of a weight matrix is one of the numbers used to judge how a layer transforms its input, and it is usually estimated rather than computed: repeated multiplication by the matrix makes any starting vector align with the leading eigenvector, after which the Rayleigh quotient gives the value.

Layer Normalisation Written as a Function (pseudocode)

Mon, 21 Sep 2026 08:15:00 +0900

Layer normalisation rescales each token’s activations using statistics computed across the features of that token alone, which is what separates it from batch normalisation: nothing in the computation depends on the other examples in the batch.

Temperature Cannot Sharpen (claim)

Mon, 21 Sep 2026 08:12:00 +0900

A claim is a statement put forward as true and argued for, not yet established as a theorem. The one here concerns temperature scaling: raising the temperature flattens a distribution, and it cannot make the most probable token more probable than it already was.

Cross-Entropy Is Convex in the Logits (property)

Mon, 21 Sep 2026 08:02:00 +0900

Convexity of a loss in the parameters is rare in deep learning, but convexity in the scores is not, and the cross-entropy of a softmax has it. The property is what guarantees that the inner optimisation — fitting the output layer, probing a frozen representation — has no local minima to fall into.

Note with Mermaid

Mon, 21 Sep 2026 07:52:00 +0900

DyCo Skills in one line: humans choose and oversee, agents draft and scale, the workflow is dynamic.

flowchart TD
A[Start] --> B{Is the chart wide?}
B -- yes --> C[Scroll horizontally]
B -- no --> D[Fit the column]
C --> E[End]
D --> E

Gradient Accumulation with an Input and Output Contract (pseudocode)

Mon, 21 Sep 2026 07:47:00 +0900

Gradient accumulation is the standard way to train with a batch larger than memory allows: several small batches are processed in turn and their gradients added before any parameter moves. The gradient of a sum is the sum of the gradients, so the result equals one step on the concatenated batch as long as the per-batch losses are averaged.

Numerically Stable Softmax in Python (starter)

Mon, 21 Sep 2026 07:42:00 +0900

A language model ends in a softmax: the raw scores of the last layer become a probability distribution over the vocabulary, and everything downstream — the training loss, sampling, beam search — reads that distribution rather than the scores themselves.

Attention Variants for Long Contexts (striped table)

Mon, 21 Sep 2026 07:32:00 +0900

Long-context attention is usually chosen on cost rather than on quality. What decides whether a variant is usable at a given sequence length is how the score cost and the per-token cache grow with \(T\), and both are easiest to compare side by side in one table rather than across several paragraphs.

Power-Law Scaling (conjecture)

Mon, 21 Sep 2026 07:22:00 +0900

A conjecture is a statement believed on the weight of evidence but not proved, and the best-known one in this field concerns how loss falls as models grow. Fits are tight over the measured range, and the conjecture is that the same form continues past it.

Notation for Multi-Head Attention (academic table)

Mon, 21 Sep 2026 07:12:00 +0900

Multi-head attention reads better as shape bookkeeping than as prose: every symbol below is a tensor, and the shapes alone decide which products are legal, which is why papers about attention variants usually open with a notation table before showing a single result.

Benchmarks for Agentic Systems (demo)

Mon, 21 Sep 2026 07:06:00 +0900

Evaluating a system that uses tools needs its own body of sources: benchmarks for tool use, for long-context reasoning and for end-to-end knowledge work, together with the model reports those numbers are read against. This note keeps that literature in a bibliography of its own, separate from the prompting and fine-tuning lists used elsewhere on this site.

A Convex Function with Many Minima (counterexample)

Mon, 21 Sep 2026 07:02:00 +0900

Convexity is often described as the property that guarantees a unique optimum, and the statement is false as written: convex functions can have a whole interval of minimisers. The counterexample is a flat-bottomed function, convex everywhere and constant on a segment.

Nucleus (Top-p) Sampling

Mon, 21 Sep 2026 06:52:00 +0900

A language model does not emit a token; it emits a probability for every token in the vocabulary, and something has to choose one. Choosing the single most probable token makes output repetitive, while sampling from the full distribution occasionally picks from its long tail of near-zero probabilities. Nucleus sampling keeps the smallest set of tokens whose combined probability reaches a threshold and samples only from those.

About