[{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nA long paper sometimes needs more than one bibliography: a main text and an appendix written against a different literature, or two halves of a review whose sources barely overlap. The theme supports that by letting each list name its own file, so each list downloads only its own entries and numbers them from one. This note is written that way on purpose — the first half is about how a model is asked, the second about how it is adapted, and the two halves share no sources.\nPart one: how a model is asked The prompting literature divides along a simple question: what is added to the prompt. The survey of the field \\cite{Yu2023-qr} organises it that way, and the empirical study of what actually matters inside a chain \\cite{Wang2022-hh} narrows the answer, arguing that the order of the steps and the presence of the question count for more than the fluency of the demonstrations. Both are worth reading before trusting a single reported gain.\nThe first family changes the shape of the prompt itself. ChainLM \\cite{Cheng2024-az} adds a verification stage so that the model checks the chain it has just written; contrastive prompting \\cite{Chia2023-cz} supplies wrong chains alongside right ones so the difference is visible; and R³ prompting \\cite{Tian2023-bk} reviews and rephrases the question before attempting it. The shared idea is that a chain is a scaffold the model can be handed, not only a trace it produces.\nThe second family removes the hand-written demonstrations altogether. Synthetic prompting \\cite{Shao2023-lx} has the model write its own examples and keeps the ones that solve the task; active prompting \\cite{Diao2024-at} selects the questions whose chains would teach the most, one round at a time; and explanation selection \\cite{Ye2023-wa} chooses, from many candidate explanations, the one that helps on unlabeled data.\nThe third family changes what happens after sampling, and that is where the reported gains are both largest and least stable. Self-consistency \\cite{Wang2022-ir} samples several chains and takes the majority answer; uncertainty-guided selection \\cite{Kumar2024-bj} decides per question whether one chain or a vote is worth the budget; the comparison of zero-shot and few-shot settings \\cite{Cheng2025-sl} finds that the ordering of methods can reverse with the prompt alone; and a study of hallucination cues \\cite{Cheng2025-wj} shows a longer chain hiding, rather than revealing, that the model is unsure. Tool-augmented chains \\cite{Inaba2023-cg} are a fourth case: the chain calls a program, which changes what an answer can be grounded in.\nReferences for the first part Part two: how a model is adapted The adaptation literature starts from a budget: what can be trained when the frozen weights are far larger than the data. Low-rank adaptation writes the change to a weight matrix as the product of two thin matrices, and PiSSA \\cite{Meng2024-pi} and CorDA \\cite{Yang2024-so} choose the subspace for those matrices from the principal directions of the pretrained weights rather than at random, reporting faster convergence at the same rank.\nA second line attacks the rank itself. Delta-LoRA \\cite{Zi2023-wp} carries the difference between successive low-rank updates into the frozen matrix, which lets the effective rank grow without training more parameters; MiLoRA \\cite{Zhang2024-xm} keeps the smallest singular directions instead of the largest, on the argument that they carry the adaptation; IncreLoRA \\cite{Zhang2023-sd} allocates rank as training proceeds, adding rows where the loss asks for them; and PRoLoRA \\cite{Sheng2024-bj} rotates the basis the update is expressed in.\nA third line is the interaction with quantisation, which is where deployment actually lands. LoftQ \\cite{Li2023-nr} initialises the low-rank pair and the quantised weights together so that the two errors cancel as far as possible; QA-LoRA \\cite{Xu2023-ro} keeps the quantised weights frozen and learns the scales that matter for the update; LQ-LoRA \\cite{Guo2023-kg} splits the problem into a low-rank part and a quantised remainder.\nA fourth line does not use a low-rank update at all. Adapters insert a bottleneck between frozen layers, and their descendants vary what is trained inside it: SparseAdapter \\cite{He2022-tw} sparsifies the bottleneck, the Hadamard adapter \\cite{Chen2024-bv} parameterises it with a fast transform, Adapter-X \\cite{Li2024-sd} generalises the idea across vision and language, and LLaMA-Adapter \\cite{Zhang2023-hq} adds a zero-initialised attention prefix that leaves the model untouched at step zero. Prompt-side methods sit at the extreme end: Prefix-tuning \\cite{Li2021-sr} trains only continuous prefix vectors, and VB-LoRA \\cite{Li2024-fk} stores many low-rank updates in a shared vector bank so that one artefact serves several tasks.\nReferences for the second part The two halves above are written against different literatures, and this note keeps them that way: the first list holds the sources of the prompting half and the second the sources of the adaptation half, each numbered from one and each fetched from its own file. A note that used a single shared bibliography would print all of them in one list, leaving the reader to work out which half of the text each entry belonged to.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-paper-with-multiple-references/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eA long paper sometimes needs more than one bibliography: a main text and an\nappendix written against a different literature, or two halves of a review whose\nsources barely overlap. The theme supports that by letting each list name its own\nfile, so each list downloads only its own entries and numbers them from one. This\nnote is written that way on purpose — the first half is about how a model is\nasked, the second about how it is adapted, and the two halves share no sources.\u003c/p\u003e","title":"A Paper with Multiple References (demo)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nA language model writes one token at a time. The prompt is processed once, then each new token is appended to the sequence and fed back in, so generation is a loop between the model and the sampler rather than a single call. The sequence diagram below shows the participants and the messages between them.\nThe loop samples from a distribution over the vocabulary, conditioned on everything generated so far: \\(x_t \\sim p(\\cdot \\mid x_{\\lt t})\\). The logits are usually divided by a temperature \\(T\\) before the softmax, and truncation (keeping only the most probable tokens) changes the support of that distribution rather than the model itself.\nsequenceDiagram autonumber participant C as Client participant M as Model participant S as Sampler C-\u0026gt;\u0026gt;M: prompt tokens x_1 .. x_n M-\u0026gt;\u0026gt;M: prefill: keys and values cached loop until stop token or token budget M-\u0026gt;\u0026gt;S: logits z_t for the next position S--\u0026gt;\u0026gt;M: sampled token x_t from softmax(z_t / T) M-\u0026gt;\u0026gt;C: stream token x_t M-\u0026gt;\u0026gt;M: append x_t and extend the cache end M-\u0026gt;\u0026gt;C: end of streamReading the diagram. The prefill call is the only one that processes the whole prompt; every later step processes one position and reuses the cached keys and values, which is why the first token arrives slowly and the rest quickly. The sampler is drawn as a separate participant because it is where generation is controlled — temperature and truncation act there, not in the model. The loop\u0026rsquo;s exit condition matters just as much: a stop token ends it early, while a token budget ends it by force, and the two produce different kinds of output.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-mermaid-sequence/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eA language model writes one token at a time. The prompt is processed once, then\neach new token is appended to the sequence and fed back in, so generation is a\nloop between the model and the sampler rather than a single call. The sequence\ndiagram below shows the participants and the messages between them.\u003c/p\u003e","title":"Autoregressive Generation (sequence diagram)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nAttention is the one place in a transformer where tokens exchange information: every other block — the projections, the activations, the normalisations — treats positions independently. That makes attention mostly shape bookkeeping, and the shapes alone decide which products are legal.\nThe code below is a module-sized step up from the previous snippet: a helper that splits the projected sequence into heads, the tensor operations of scaled dot-product attention, an assertion that the head arithmetic divides evenly, and a property check at the bottom. The mask is what makes the block usable as a next-token predictor: position \\(t\\) may look at positions up to \\(t\\) and nothing later.\nimport numpy as np def split_heads(x, n_heads, d_head): \u0026#34;\u0026#34;\u0026#34;(B, T, d_model) -\u0026gt; (B, H, T, d_head).\u0026#34;\u0026#34;\u0026#34; B, T, _ = x.shape return x.reshape(B, T, n_heads, d_head).transpose(0, 2, 1, 3) def causal_attention(X, Wq, Wk, Wv, n_heads): \u0026#34;\u0026#34;\u0026#34;Multi-head self-attention over a batch of sequences. X: (B, T, d_model) Wq, Wk, Wv: (d_model, d_model) \u0026#34;\u0026#34;\u0026#34; B, T, d_model = X.shape assert d_model % n_heads == 0, \u0026#34;d_model must divide by n_heads\u0026#34; d_head = d_model // n_heads Q = split_heads(X @ Wq, n_heads, d_head) # (B, H, T, d_head) K = split_heads(X @ Wk, n_heads, d_head) V = split_heads(X @ Wv, n_heads, d_head) scores = Q @ K.transpose(0, 1, 3, 2) / np.sqrt(d_head) # (B,H,T,T) future = np.triu(np.ones((T, T), dtype=bool), k=1) scores = np.where(future, -np.inf, scores) weights = np.exp(scores - scores.max(axis=-1, keepdims=True)) weights /= weights.sum(axis=-1, keepdims=True) # rows sum to 1 out = weights @ V # (B, H, T, d_head) return out.transpose(0, 2, 1, 3).reshape(B, T, d_model) if __name__ == \u0026#34;__main__\u0026#34;: rng = np.random.default_rng(0) B, T, d_model, n_heads = 2, 5, 16, 4 X = rng.normal(size=(B, T, d_model)) W = rng.normal(size=(d_model, d_model)) / np.sqrt(d_model) P = causal_attention(X, W, W, W, n_heads) print(\u0026#34;output shape\u0026#34;, P.shape) # A position may not affect the ones before it: change the last token # and check that every earlier output is unchanged. X2 = X.copy() X2[:, -1] += 1.0 P2 = causal_attention(X2, W, W, W, n_heads) same = np.allclose(P[:, :-1], P2[:, :-1]) print(\u0026#34;earlier positions unchanged:\u0026#34;, same) Run as printed, it reports the output shape (2, 5, 16) and earlier positions unchanged: True. The check perturbs the last token and confirms that no earlier output moves, which is the property the mask exists to guarantee. The scores matrix costs \\(T^2\\) entries per head, which is why long-context variants trade that term away.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-causal-attention-python/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eAttention is the one place in a transformer where tokens exchange information:\nevery other block — the projections, the activations, the normalisations —\ntreats positions independently. That makes attention mostly shape bookkeeping,\nand the shapes alone decide which products are legal.\u003c/p\u003e","title":"Causal Multi-Head Attention in Python (intermediate)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nThe smallest interesting attention example has two positions and one head, which is enough for a full calculation with no matrix product larger than two by two.\nWorking it at this size makes the two roles visible: the score decides how much the output at position two borrows from position one, and the value vectors set what is borrowed. With a causal mask, the first row has nothing to attend to but itself.\nExample. Let \\(d_h = 2\\) and take\n$$ Q = K = \\begin{pmatrix} 1 \u0026 0 \\\\ 0 \u0026 1 \\end{pmatrix}, \\qquad V = \\begin{pmatrix} 1 \u0026 0 \\\\ 0 \u0026 2 \\end{pmatrix} . $$The unnormalised scores are \\(QK^{\\top} = I\\), so with \\(\\sqrt{d_h} = \\sqrt{2}\\) the scaled scores are the identity divided by \\(1.414\\): \\(0.707\\) on the diagonal and zero elsewhere. After the causal mask the upper-right entry is set to \\(-\\infty\\), giving\n$$ S_{\\text{masked}} = \\begin{pmatrix} 0.707 \u0026 -\\infty \\\\ 0 \u0026 0.707 \\end{pmatrix}, \\qquad A = \\begin{pmatrix} 1 \u0026 0 \\\\ 0.354 \u0026 0.646 \\end{pmatrix}, $$where the second row of \\(A\\) splits its weight between the two positions because the two scores there are \\(0\\) and \\(0.707\\). The output is then\n$$ A V = \\begin{pmatrix} 1 \u0026 0 \\\\ 0.354 \u0026 1.293 \\end{pmatrix} . $$Row one returns the first value vector unchanged — it had nowhere else to look.\nThe example is worth keeping in mind when reading an attention map: a row that concentrates on one position is not necessarily a strong match, since the causal mask forces the first row into that shape regardless of the scores.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-example-attention-two-tokens/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eThe smallest interesting attention example has two positions and one head, which is\nenough for a full calculation with no matrix product larger than two by two.\u003c/p\u003e\n\u003cp\u003eWorking it at this size makes the two roles visible: the score decides how much the output\nat position two borrows from position one, and the value vectors set what is borrowed. With\na causal mask, the first row has nothing to attend to but itself.\u003c/p\u003e","title":"Attention with Two Tokens (example)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nA project that produces a paper has to keep three kinds of thing apart that are easy to mix: the installed source, the outputs of individual runs, and the material the paper is written from. Once several sweeps and a performance benchmark live in the same repository, the folder structure is what decides whether a figure can be traced back to the run that produced it.\nThe tree below is five levels deep in places. Source sits under src/ as an installable package, each run gets its own timestamped folder under experiments/runs/, benchmark scripts and their results live together, and the paper\u0026rsquo;s figures and tables are generated from the run folders by a script rather than pasted in by hand.\nscaling-study/ ├── pyproject.toml ├── src/ │ └── scaling_study/ │ ├── data/ │ │ ├── tokenizer.py # train or load the tokenizer │ │ └── shards.py # memory-mapped token shards │ ├── model/ │ │ ├── attention.py # causal attention │ │ ├── blocks.py # residual + norm + MLP │ │ └── config.py # dataclass per model size │ ├── kernels/ │ │ ├── reference.py # slow, obviously correct │ │ └── triton/ # fused attention kernels │ ├── train/ │ │ ├── loop.py # one step, one eval, one checkpoint │ │ ├── optim.py # schedule and gradient clipping │ │ └── distribute.py # data parallel wrapper │ └── eval/ │ ├── perplexity.py # loss per token, not per sequence │ └── scaling_fit.py # power-law fit over sizes ├── configs/ │ ├── registry.py # names the sweeps │ └── sweep/ # one file per sweep point ├── experiments/ │ └── runs/ │ └── 2026-09-18-a3f2/ │ ├── config.yaml # exactly what was launched │ ├── metrics.jsonl # one line per evaluation │ └── checkpoints/ ├── benchmarks/ │ ├── attention_bench.py │ └── results/ ├── paper/ │ ├── make_figures.py # reads runs/, writes figures │ ├── figures/ │ └── tables/ └── tests/ The folder that carries the most weight is experiments/runs/: one directory per launch, holding the config it was started with, its metric log and its checkpoints. That single convention is what turns a plot into something reproducible — a figure script can walk those directories, read the loss curves and rebuild every panel from artifacts, so a reviewer\u0026rsquo;s question about which configuration produced a point has a file to look at rather than a memory to trust.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-folder-structure-advanced/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eA project that produces a paper has to keep three kinds of thing apart that are\neasy to mix: the installed source, the outputs of individual runs, and the\nmaterial the paper is written from. Once several sweeps and a performance\nbenchmark live in the same repository, the folder structure is what decides\nwhether a figure can be traced back to the run that produced it.\u003c/p\u003e","title":"Layout of a Scaling Study (advanced folder structure)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nChain-of-thought prompting began as a single sentence appended to a prompt and is now a family of strategies that differ in what they add: a fixed reasoning skeleton, several sampled chains and a vote, demonstrations the method writes for itself, or calls to an external tool in the middle of the reasoning. This survey groups the entries in its bibliography by that question, following the framing of the earlier survey of prompting strategies \\cite{Yu2023-qr}, and asks in each group what the reported gain is actually attributed to.\nTable 1. The families differ in what they add to the prompt and in what the reported improvement is attributed to.\nFamily What is added Reported effect Representative work Single chain an instruction to reason step by step large gains on multi-step tasks \\cite{Wang2022-hh}, \\cite{Cheng2025-sl} Structured chain a fixed order of sub-tasks fewer unsupported statements \\cite{Sultan2024-rc}, \\cite{Cheng2024-az} Sampling and voting several chains, then an aggregation rule accuracy rises with the sample count self-consistency \\cite{Wang2022-ir}, uncertainty-guided \\cite{Kumar2024-bj} Demonstration construction chains written or selected by the method removes hand-written examples \\cite{Shao2023-lx}, \\cite{Diao2024-at}, \\cite{Ye2023-wa} Contrastive and corrective wrong chains as negative examples fewer shortcut answers \\cite{Chia2023-cz}, \\cite{Tian2023-bk} Tool-augmented calls to an external tool inside the chain grounding on computation \\cite{Inaba2023-cg} The diagram below shows where the families differ in time: the first four change what is put into the prompt, while sampling and voting and the tool-augmented family change what happens after the model has answered.\nsequenceDiagram participant Q as Question participant P as Prompt participant M as Model participant A as Aggregator Q-\u0026gt;\u0026gt;P: question plus worked examples P-\u0026gt;\u0026gt;M: ask for k independent chains M-\u0026gt;\u0026gt;M: reason step by step M-\u0026gt;\u0026gt;A: k answers with their chains A-\u0026gt;\u0026gt;A: majority vote over answers A-\u0026gt;\u0026gt;Q: single answerProposed theory: reliability of a chain and the gain from voting. Model a chain of \\(L\\) steps as correct only if every step is, with per-step reliability \\(p\\) and independence between steps, and write \\(q\\) for the probability that a single sampled chain yields the correct final answer. The theory this survey proposes for testing is the pair of statements below: the reliability of one chain, and what voting over several of them does to it.\n$$ \\Pr(\\text{chain is correct}) = p^{L} , $$$$ \\Pr(\\text{majority vote is correct}) \\to 1 \\quad \\text{as} \\ k \\to \\infty \\qquad \\text{whenever} \\quad q \\gt \\tfrac{1}{2} . $$The second statement is where the method\u0026rsquo;s limit lives: sampling changes neither \\(p\\) nor \\(L\\), so at \\(q \\leq \\tfrac{1}{2}\\) more samples do not help, and a task on which chains are individually unreliable needs a change to the prompt rather than a larger sample.\nAlgorithm 1 writes the sampling rule that the voting family shares, and makes the two quantities above concrete: \\(k\\) independent draws and one aggregation step.\n\\begin{algorithm}[H] \\footnotesize \\caption{Self-consistency decoding} \\begin{algorithmic}[1] \\Require question \\(x\\), prompt \\(P\\), sample count \\(k\\), temperature \\(T\\) \\State draw \\(k\\) chains independently from \\(p_\\theta(\\cdot \\mid P, x)\\) at temperature \\(T\\) \\For{each sampled chain} \\State extract the final answer \\Comment{string match, or a task-specific parser} \\EndFor \\State return the most frequent answer, with its share of the vote \\end{algorithmic} \\end{algorithm} Listing 1 is the aggregation step on its own, which is where the method is usually implemented badly: the vote is over extracted answers, not over chains, so an answer that appears in fluent prose has to be parsed before it can be counted. Run as printed it reports answer B with 57% of the vote.\nfrom collections import Counter def self_consistency(answers): \u0026#34;\u0026#34;\u0026#34;Majority vote over the answers extracted from sampled chains.\u0026#34;\u0026#34;\u0026#34; counts = Counter(answers) best, votes = counts.most_common(1)[0] return best, votes / len(answers) if __name__ == \u0026#34;__main__\u0026#34;: answers = [\u0026#34;B\u0026#34;, \u0026#34;C\u0026#34;, \u0026#34;B\u0026#34;, \u0026#34;B\u0026#34;, \u0026#34;D\u0026#34;, \u0026#34;C\u0026#34;, \u0026#34;B\u0026#34;] answer, share = self_consistency(answers) print(f\u0026#34;answer {answer} with {share:.0%} of the vote\u0026#34;) What the surveyed work leaves open is the same in both directions: whether a chain improves an answer or merely accompanies it, which the studies of hallucination cues \\cite{Cheng2025-wj} and of what matters in a chain \\cite{Wang2022-hh} both probe, and how a demonstrated gain should be reported when the same prompt is reused across tasks — the question the earlier survey raises and which the later comparisons of zero-shot and few-shot settings \\cite{Cheng2025-sl} make sharper.\nReferences ","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-survey-chain-of-thought/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eChain-of-thought prompting began as a single sentence appended to a prompt and is\nnow a family of strategies that differ in what they add: a fixed reasoning\nskeleton, several sampled chains and a vote, demonstrations the method writes for\nitself, or calls to an external tool in the middle of the reasoning. This survey\ngroups the entries in its bibliography by that question, following the framing of\nthe earlier survey of prompting strategies \\cite{Yu2023-qr}, and asks in each\ngroup what the reported gain is actually attributed to.\u003c/p\u003e","title":"A Survey of Chain-of-Thought Prompting: Single Chains, Sampling and Demonstration Construction (survey)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nAn identity is an equation that holds for every admissible value of its variables, and this one converts a variance into two expectations, which is often the difference between a quantity that can be estimated in one pass and one that cannot.\nIts use in this field is everywhere a second moment appears: batch normalisation, the analysis of gradient noise, and the decomposition of an estimator\u0026rsquo;s error all begin by applying it.\nIdentity (variance). For a random variable \\(X\\) with finite second moment,\n$$ \\operatorname{Var}(X) = \\mathbb{E}\\!\\left[X^{2}\\right] - \\left(\\mathbb{E}[X]\\right)^{2}, \\qquad \\operatorname{Var}(X) = \\mathbb{E}\\!\\left[(X - \\mathbb{E}[X])^{2}\\right] . $$The equality follows by expanding the square in the second expression and using linearity of expectation. The practical difference between the two forms is what has to be stored: the first can be accumulated from running sums of \\(x\\) and \\(x^{2}\\), while the second wants the mean before the sum can be formed — a difference that matters when statistics are computed over a stream.\nRead as a constraint rather than a formula, the identity says a variance can never be negative, since it is an expectation of a square; the first form makes that fact easy to lose sight of, and the second makes it immediate.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-identity-variance/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eAn identity is an equation that holds for every admissible value of its variables, and\nthis one converts a variance into two expectations, which is often the difference between\na quantity that can be estimated in one pass and one that cannot.\u003c/p\u003e","title":"The Variance Identity (identity)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nBayes\u0026rsquo; rule is the rule for turning a model\u0026rsquo;s likelihood into a statement about hypotheses, and it is used twice in a language-model workflow: once to reason about uncertainty in evaluation, and once whenever a prior over prompts or parameters is combined with observed data.\nWritten out, the rule is a rearrangement of the definition of conditional probability, and its content is the division by the evidence term, which is what makes the posterior a distribution rather than a score.\nRule (Bayes). For events \\(A\\) and \\(B\\) with \\(\\Pr(B) \\gt 0\\),\n$$ \\Pr(A \\mid B) = \\frac{\\Pr(B \\mid A)\\,\\Pr(A)}{\\Pr(B)} , \\qquad \\Pr(B) = \\sum_{A'} \\Pr(B \\mid A')\\,\\Pr(A') , $$and in the continuous case with densities the sums become integrals. The three ingredients are the prior \\(\\Pr(A)\\), the likelihood \\(\\Pr(B \\mid A)\\) and the evidence \\(\\Pr(B)\\), and only the first two are modelling choices: the evidence is whatever makes the posterior integrate to one.\nUsing the rule well means noticing when the evidence is hard to compute: in models with many parameters the denominator is an integral over everything, which is why the posterior is usually approximated rather than evaluated.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-rule-bayes/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eBayes\u0026rsquo; rule is the rule for turning a model\u0026rsquo;s likelihood into a statement about\nhypotheses, and it is used twice in a language-model workflow: once to reason about\nuncertainty in evaluation, and once whenever a prior over prompts or parameters is\ncombined with observed data.\u003c/p\u003e","title":"Bayes' Rule (rule)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nA rule is a procedure applied so often that it stops looking like a theorem, and the chain rule is the extreme case: every backward pass through a stacked model is one long application of it.\nStated for compositions of two functions and applied left to right through a network, it converts the derivative of the loss with respect to the inputs into a product of local derivatives, each computed at the activations that were stored during the forward pass.\nRule (chain rule). If \\(f\\) is differentiable at \\(g(x)\\) and \\(g\\) is differentiable at \\(x\\), then the composition is differentiable and\n$$ \\frac{d}{dx} f(g(x)) = f'(g(x)) \\, g'(x). $$For \\(n\\) composed functions \\(y = f_n(f_{n-1}(\\cdots f_1(x)\\cdots))\\) the rule is applied in turn, giving\n$$ \\frac{dy}{dx} = \\prod_{k=1}^{n} f_k'\\!\\left(f_{k-1}(\\cdots f_1(x)\\cdots)\\right). $$Each factor is evaluated at the intermediate value the forward pass produced, which is why backpropagation stores activations rather than recomputing them.\nThe product form also explains a failure mode: when the factors are consistently below one in magnitude, the product shrinks geometrically with depth, which is the mechanism behind vanishing gradients in a recurrent or very deep model.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-rule-chain/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eA rule is a procedure applied so often that it stops looking like a theorem, and the\nchain rule is the extreme case: every backward pass through a stacked model is one long\napplication of it.\u003c/p\u003e","title":"The Chain Rule (rule)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nThe smallest useful project around a piece of neural-network mathematics holds one idea per file and keeps the generated numbers out of the code. Nothing in it is conventional yet: a starter folder is decided by what you want to compare next, not by what a template says.\nThe tree below is that starter size — a few files at the top level and one folder for data. Each entry carries a short note after #, and the highlighter treats those as comments, so the structure reads first and the explanations stay out of the way.\nsoftmax-lab/ ├── data/ │ └── logits.npy # one batch of scores to experiment with ├── softmax.py # the stable version: max-shift, then exp ├── temperature.py # the same softmax, with a temperature ├── check_overflow.py # naive vs shifted, on scores up to 1e3 └── README.md Keeping the generated data/ folder separate from the scripts is the one habit worth adopting at this size. It is also the habit that runs out first: once a second model variant arrives, the flat layout forces the choice between longer file names and imports that depend on where the script was launched from — the reason the next size up introduces folders.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-folder-structure-starter/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eThe smallest useful project around a piece of neural-network mathematics holds\none idea per file and keeps the generated numbers out of the code. Nothing in it\nis conventional yet: a starter folder is decided by what you want to compare\nnext, not by what a template says.\u003c/p\u003e","title":"Layout of a Small Numeric Experiment (starter folder structure)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nBackpropagation computes, for one layer, three gradients: one for each parameter of the layer and one to hand to the layer before it. The third is what makes deep networks trainable — without it, error could not travel past the last layer. The procedure below is the backward pass of a single dense layer.\nThe arithmetic follows from the chain rule. For \\(Y = XW + b\\), the gradient with respect to the weights contracts the input with the incoming gradient, \\(\\partial L/\\partial W = X^{\\top}\\,\\partial L/\\partial Y\\), while the bias gradient is the incoming gradient summed over the batch. The gradient passed backwards multiplies by the transposed weights, \\(\\partial L/\\partial X = \\partial L/\\partial Y\\, W^{\\top}\\), so each layer receives exactly the shape it produced during the forward pass.\n\\begin{algorithm}[H] \\caption{Backward Pass Through a Dense Layer} \\begin{algorithmic}[1] \\Require Layer input \\(X \\in \\mathbb{R}^{n \\times d_{in}}\\) \\Require Weights \\(W \\in \\mathbb{R}^{d_{in} \\times d_{out}}\\), bias \\(b \\in \\mathbb{R}^{d_{out}}\\) \\Require Upstream gradient \\(\\frac{\\partial L}{\\partial Y} \\in \\mathbb{R}^{n \\times d_{out}}\\) \\Ensure Parameter gradients \\(\\frac{\\partial L}{\\partial W}\\), \\(\\frac{\\partial L}{\\partial b}\\) and the gradient \\(\\frac{\\partial L}{\\partial X}\\) \\State Recover the cached forward quantity: \\(Y \\leftarrow XW + b\\) \\State Weight gradient: contract the input with the incoming gradient \\State \\hspace{0.5cm} \\(\\frac{\\partial L}{\\partial W} \\leftarrow X^{\\top} \\frac{\\partial L}{\\partial Y}\\) \\State Bias gradient: sum the incoming gradient over the batch \\State \\hspace{0.5cm} \\(\\frac{\\partial L}{\\partial b} \\leftarrow \\sum_{i=1}^{n} \\frac{\\partial L}{\\partial Y}[i, :]\\) \\State Gradient for the previous layer: apply the chain rule once more \\State \\hspace{0.5cm} \\(\\frac{\\partial L}{\\partial X} \\leftarrow \\frac{\\partial L}{\\partial Y} W^{\\top}\\) \\Return \\(\\frac{\\partial L}{\\partial W}\\), \\(\\frac{\\partial L}{\\partial b}\\), \\(\\frac{\\partial L}{\\partial X}\\) \\end{algorithmic} \\end{algorithm} Reading the algorithm. Step 1 is a reminder that the forward quantity must have been cached: the backward pass reuses what the forward pass computed, which is why training needs memory proportional to depth. Steps 2–3 are two matrix multiplications — the same cost as the forward pass — and step 4 is the part that propagates error to earlier layers. Note the asymmetry with the forward pass: the forward pass multiplies by \\(W\\), the backward pass by \\(W^{\\top}\\), and it is that transposition which keeps the shapes consistent in both directions.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-backpropagation-pseudocode/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eBackpropagation computes, for one layer, three gradients: one for each parameter\nof the layer and one to hand to the layer before it. The third is what makes deep\nnetworks trainable — without it, error could not travel past the last layer. The\nprocedure below is the backward pass of a single dense layer.\u003c/p\u003e","title":"Backpropagation Through a Dense Layer"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nAttention lets a model decide, for each position it is processing, which other positions to draw information from. This note states the scaled dot-product form as an algorithm rather than as an equation, so the sequence of operations — and the shape of each intermediate result — is explicit.\nThe scaling by \\( \\sqrt{d_k} \\) matters: without it, the dot products grow with the key dimension, the softmax saturates, and the weights collapse onto a single position.\n\\begin{algorithm}[H] \\caption{Scaled Dot-Product Attention} \\begin{algorithmic}[1] \\Require Query matrix \\(Q \\in \\mathbb{R}^{n \\times d_k}\\) \\Require Key matrix \\(K \\in \\mathbb{R}^{m \\times d_k}\\) \\Require Value matrix \\(V \\in \\mathbb{R}^{m \\times d_v}\\) \\Ensure Weighted output \\(O \\in \\mathbb{R}^{n \\times d_v}\\) \\State Compute the similarity between every query and every key: \\State \\hspace{0.5cm} \\(S \\leftarrow Q K^{\\top}\\) with \\(S \\in \\mathbb{R}^{n \\times m}\\) \\State Scale the scores to keep the softmax in a stable range: \\State \\hspace{0.5cm} \\(S \\leftarrow S / \\sqrt{d_k}\\) \\State Convert scores into weights that sum to one over the keys: \\State \\hspace{0.5cm} \\(A \\leftarrow \\mathrm{softmax}(S)\\) rowwise \\State Read out a weighted combination of the values: \\State \\hspace{0.5cm} \\(O \\leftarrow A V\\) \\Return \\(O\\) \\end{algorithmic} \\end{algorithm} Reading the algorithm. Steps 1–2 build the score matrix, where row \\(i\\) holds the compatibility of query \\(i\\) with every key; step 3 turns each row into a probability distribution, so a high score for one key means its value dominates that output row; step 4 mixes the values accordingly. Every row of the output therefore draws on a different combination of the same value matrix, which is what makes attention content-adaptive rather than fixed — at a cost of \\(O(nm)\\) for forming \\(S\\), the pressure behind the approximate and sparse variants.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-attention-pseudocode/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eAttention lets a model decide, for each position it is processing, which\nother positions to draw information from. This note states the \u003cstrong\u003escaled\ndot-product\u003c/strong\u003e form as an algorithm rather than as an equation, so the\nsequence of operations — and the shape of each intermediate result — is\nexplicit.\u003c/p\u003e","title":"Attention Mechanism: Pseudocode"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nAttention is built on a quantity that looks like a similarity but is defined by three axioms: an inner product is symmetric, linear in each argument, and positive definite. Those three properties are what allow a score to be read as \u0026ldquo;how much one vector points along another\u0026rdquo;.\nWriting the axioms down is not pedantry here. The positive-definiteness clause is the one that rules out degenerate score functions, and the linearity clauses are what let the whole score matrix be computed as one matrix product.\nAxiom (inner product). A function \\(\\langle \\cdot, \\cdot \\rangle\\) on two vectors is an inner product when\n$$ \\langle x, y \\rangle = \\langle y, x \\rangle, \\qquad \\langle \\alpha x + \\beta z, y \\rangle = \\alpha \\langle x, y \\rangle + \\beta \\langle z, y \\rangle , $$for all vectors and scalars, together with \\(\\langle x, x \\rangle \\gt 0\\) for every non-zero \\(x\\). The score \\(q^{\\top} k\\) satisfies all three, which is why attention is written as a product of matrices rather than as a table of pairwise scores.\nOnly the first two axioms are used when the scores are computed; positive definiteness is what guarantees that a vector has positive length, and it is invoked whenever a normalisation or a bound relies on \\(\\lVert x \\rVert \\gt 0\\) for \\(x \\neq 0\\).\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-axiom-inner-product/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eAttention is built on a quantity that looks like a similarity but is defined by three\naxioms: an inner product is symmetric, linear in each argument, and positive definite.\nThose three properties are what allow a score to be read as \u0026ldquo;how much one vector points\nalong another\u0026rdquo;.\u003c/p\u003e","title":"Inner-Product Axioms (axiom)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nPosition information has to enter attention somewhere, and one way is to rotate the query and key vectors by a position-dependent angle. The formula is appealing because the effect on the score depends only on the difference of the two positions.\nThat property is what the notation is chosen to display: a rotation preserves lengths and turns an inner product between rotated vectors into a function of the angle between them.\nFormula (rotary embedding). Split a head vector \\(x \\in \\mathbb{R}^{d_h}\\) into \\(d_h/2\\) pairs \\((x_{2i-1}, x_{2i})\\). At position \\(m\\) and frequency \\(\\theta_i\\), the rotated vector \\(R_{\\Theta, m} x\\) has, in each pair,\n$$ \\begin{pmatrix} \\tilde{x}_{2i-1} \\\\ \\tilde{x}_{2i} \\end{pmatrix} = \\begin{pmatrix} \\cos(m\\theta_i) \u0026 -\\sin(m\\theta_i) \\\\ \\sin(m\\theta_i) \u0026 \\cos(m\\theta_i) \\end{pmatrix} \\begin{pmatrix} x_{2i-1} \\\\ x_{2i} \\end{pmatrix}, \\qquad \\theta_i = 10000^{-2(i-1)/d_h} . $$Rotating both the query at position \\(m\\) and the key at position \\(n\\) gives an inner product that depends on \\(m - n\\) rather than on either position alone, because a rotation by \\(m\\theta\\) composed with the inverse rotation by \\(n\\theta\\) is a rotation by \\((m-n)\\theta\\).\nThe price of the construction is that it encodes relative position only through rotation: a query and a key at the same position are unaffected, so the scheme says nothing about absolute location, which is either a feature or a limitation depending on the task.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-formula-rope/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003ePosition information has to enter attention somewhere, and one way is to rotate the query\nand key vectors by a position-dependent angle. The formula is appealing because the effect\non the score depends only on the difference of the two positions.\u003c/p\u003e","title":"Rotary Position Embedding (formula)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nA scaling study compares models of different sizes trained on the same token budget. If the loss falls smoothly as capacity grows, the family is in a regime where parameters still buy accuracy, and the fall can be summarised as a power law in the parameter count rather than reported as a list of separate runs.\nThe table below is the theme\u0026rsquo;s default table style: every cell carries a border, the header row is tinted, and the numeric columns are right-aligned so that digits line up vertically. That alignment comes from the delimiter row of the Markdown table, not from the stylesheet, so an author can set it per column without touching any CSS.\nParameters \\(d_{\\text{model}}\\) Heads Validation loss Perplexity 25M 384 6 3.412 30.3 50M 512 8 3.128 22.8 125M 768 12 2.914 18.4 350M 1024 16 2.703 14.9 760M 1280 20 2.561 12.9 Fitting the last four rows gives a loss that decays roughly as \\(N^{-0.07}\\), near the exponent reported for larger model families, so the shape of the curve is the expected one. The absolute numbers are not: the runs behind this demo used a fraction of the usual token budget, which is why the closing paragraph of a real note would carry the run configuration and the seed count instead.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-loss-by-model-size-table/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eA scaling study compares models of different sizes trained on the same token\nbudget. If the loss falls smoothly as capacity grows, the family is in a regime\nwhere parameters still buy accuracy, and the fall can be summarised as a power\nlaw in the parameter count rather than reported as a list of separate runs.\u003c/p\u003e","title":"Validation Loss by Model Size (bordered table)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nAn algorithm is a procedure with a specification, and this one says how the attention definition is executed on a machine: scale the scores, remove the positions that are not allowed, normalise and mix.\nThe steps are written in the order they must be performed. Masking before the softmax is the detail that matters — doing it afterwards would leave the removed positions contributing to the normalising constant and change the result.\nAlgorithm (masked softmax attention). Given \\(Q, K, V \\in \\mathbb{R}^{T \\times d_h}\\) and a boolean mask \\(M\\), the steps are:\nCompute the scaled scores \\(S = QK^{\\top} / \\sqrt{d_h}\\), of shape \\(T \\times T\\). Replace the scores that are not allowed to be attended to with \\(-\\infty\\), that is \\(S_{ij} \\leftarrow -\\infty\\) wherever \\(M_{ij}\\) is true. Subtract the row maximum: \\(S_{ij} \\leftarrow S_{ij} - \\max_k S_{ik}\\), which leaves the softmax unchanged and keeps the exponentials finite. Exponentiate and normalise each row: \\(A_{ij} = e^{S_{ij}} / \\sum_k e^{S_{ik}}\\). Return \\(A V\\). The cost is dominated by step one, which forms a \\(T \\times T\\) matrix per head and therefore grows quadratically in the sequence length. Steps three and four are the numerical stabilisation from the log-sum-exp bounds.\nTwo of the five steps are present for reasons other than the mathematics: masking before normalising keeps the definition exact, and subtracting the row maximum keeps the arithmetic finite — the algorithm as written and the definition as stated compute the same function, and only one of the two is executable.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-algorithm-masked-attention/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eAn algorithm is a procedure with a specification, and this one says how the attention\ndefinition is executed on a machine: scale the scores, remove the positions that are not\nallowed, normalise and mix.\u003c/p\u003e","title":"Masked Softmax Attention (algorithm)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nThe Woodbury identity rewrites the inverse of a matrix plus a low-rank correction in terms of the inverse of the original matrix and a much smaller inverse. It is the algebraic fact behind every method that updates a large system cheaply.\nThe sizes are what makes it useful: the correction inverse is of the order of the rank rather than of the matrix, so an update of rank \\(r\\) costs \\(r^{3}\\) work instead of \\(n^{3}\\).\nIdentity (Woodbury). Let \\(A \\in \\mathbb{R}^{n \\times n}\\) be invertible, and let \\(U \\in \\mathbb{R}^{n \\times r}\\), \\(C \\in \\mathbb{R}^{r \\times r}\\) and \\(V \\in \\mathbb{R}^{r \\times n}\\) with \\(C\\) and \\(A + UCV\\) invertible. Then\n$$ \\left(A + UCV\\right)^{-1} = A^{-1} - A^{-1} U \\left(C^{-1} + V A^{-1} U\\right)^{-1} V A^{-1}, $$where the middle inverse is \\(r \\times r\\). The identity is verified by multiplying the right-hand side by \\(A + UCV\\) and collecting terms; the special case \\(C = I\\), \\(V = U^{\\top}\\) gives the update used when a low-rank term is added to a covariance.\nIn a model, the pattern shows up whenever a large linear system is modified by something thin — a new adapter, a few constrained directions, a rank-limited edit — since the identity turns the cost of the update from the size of the matrix into the size of the change.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-identity-woodbury/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eThe Woodbury identity rewrites the inverse of a matrix plus a low-rank correction in\nterms of the inverse of the original matrix and a much smaller inverse. It is the algebraic\nfact behind every method that updates a large system cheaply.\u003c/p\u003e","title":"The Woodbury Identity (identity)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nThe linear representation hypothesis conjectures that the concepts a network has learned are encoded as directions in activation space, so that adding a concept vector to an activation moves the computation in a meaningful way.\nThe evidence is a body of interventions that behave as the conjecture predicts: a single direction can be amplified or suppressed with roughly monotone effects on the output, and differences between activations carry information the probe can read linearly.\nConjecture (linear representation). For a concept \\(c\\) there exists a direction \\(u_c\\) in the residual stream such that changing an activation \\(h\\) to \\(h + \\lambda u_c\\) changes the model\u0026rsquo;s behaviour with respect to \\(c\\) in a way that is monotone in \\(\\lambda\\), and such that the projection \\(u_c^{\\top} h\\) predicts the presence of the concept across contexts.\nStated in that form the conjecture is falsifiable in the way a mathematical statement should be: it predicts that the projection is a sufficient statistic for the concept, which is a claim a probe can test against held-out contexts.\nThe conjecture is used constantly in practice — steering vectors, concept erasure and activation arithmetic all assume it — which is a reminder that a useful assumption does not have to be a proved one.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-conjecture-features-directions/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eThe linear representation hypothesis conjectures that the concepts a network has\nlearned are encoded as directions in activation space, so that adding a concept vector to\nan activation moves the computation in a meaningful way.\u003c/p\u003e","title":"Features Are Directions (conjecture)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nCaching keys and values is standard practice, and the claim that justifies it is strong: incremental decoding gives exactly the same outputs as recomputing attention over the whole sequence, not an approximation of them.\nStated for the arithmetic of the model rather than for floating-point hardware, the claim is what licenses treating the cache as an optimisation. The caveat about floating point is worth stating alongside it, because it is where a literal reading of the claim fails.\nClaim. Let a decoder attend over a prefix of length \\(t\\), and let the same decoder attend over the prefix extended to length \\(t + 1\\) with the additional key \\(k_{t+1}\\) and value \\(v_{t+1}\\) appended to the cached matrices. Then\n$$ \\operatorname{softmax}\\!\\left(\\frac{q_{t+1} K_{1:t+1}^{\\top}}{\\sqrt{d_h}}\\right) V_{1:t+1} = \\left[\\operatorname{softmax}\\!\\left(\\frac{Q_{1:t+1} K_{1:t+1}^{\\top}}{\\sqrt{d_h}}\\right) V_{1:t+1}\\right]_{t+1} $$the left side being the incremental step that uses only the cached keys and values and the current query, and the right side being row \\(t + 1\\) of the full computation over the whole prefix. The equality holds because the cache stores exactly the keys and values the full computation would use, and the softmax at a causal position depends only on those and on the current query.\nThe claim is about the mathematical function, and it is exact there; in floating point the two orders of summation can differ in the last bits, which is why a cache and a recomputation may differ in the eighth decimal place while computing the same mathematical quantity.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-claim-kv-cache-exact/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eCaching keys and values is standard practice, and the claim that justifies it is strong:\nincremental decoding gives exactly the same outputs as recomputing attention over the whole\nsequence, not an approximation of them.\u003c/p\u003e","title":"The KV Cache Is Exact (claim)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nChoosing an attention variant is a two-constraint problem rather than a ranking: the score cost decides how long a sequence can be trained on, and the per-token cache decides how long a sequence can be served, and the two orderings of the variants are not the same.\nThis note uses a quadrant chart, which is the right shape for two independent axes with no third quantity to show. Positions are qualitative — ranked rather than measured — and the axes run from cheap to expensive on both counts.\nquadrantChart title Attention variants by score cost and cache size x-axis Cheap scores --\u0026gt; Expensive scores y-axis Small cache --\u0026gt; Large cache quadrant-1 Expensive, large cache quadrant-2 Expensive, small cache quadrant-3 Cheap, small cache quadrant-4 Cheap, large cache Full attention: [0.9, 0.85] Sliding window: [0.45, 0.8] Block sparse: [0.62, 0.58] Kernel linear: [0.2, 0.18] State space: [0.25, 0.1]The chart makes the usual trade-off visible in one picture: every variant that reduces the score cost by sparsifying still pays the full cache, so it lands in the right-hand column, while the variants that replace the pairwise structure move to the bottom left and pay instead in expressiveness.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-mermaid-quadrant-attention/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eChoosing an attention variant is a two-constraint problem rather than a ranking:\nthe score cost decides how long a sequence can be trained on, and the per-token\ncache decides how long a sequence can be served, and the two orderings of the\nvariants are not the same.\u003c/p\u003e","title":"Attention Variants by Cost and Cache (quadrant chart)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nOne training step is a fixed cycle: the model predicts, the predictions are scored against the targets, the error is propagated backwards, and the parameters move a little in the direction that reduces it. The diagram below shows that cycle; the repetition is the whole of training.\nIn notation, the step computes a loss \\(L(\\theta)\\) over a batch, obtains the gradient \\(\\nabla_\\theta L\\) by backpropagation, and applies an update \\(\\theta \\leftarrow \\theta - \\eta \\nabla_\\theta L\\), where \\(\\eta\\) is the learning rate. Adaptive optimisers such as Adam rescale that update per parameter, but the loop structure is unchanged.\nflowchart TD A[\u0026#34;Input batch: tokens X, targets Y\u0026#34;] --\u0026gt; B[\u0026#34;Forward pass: logits Z = f_theta(X)\u0026#34;] B --\u0026gt; C[\u0026#34;Loss: L = -sum log p_theta(Y | X)\u0026#34;] C --\u0026gt; D[\u0026#34;Backward pass: gradients of L wrt theta\u0026#34;] D --\u0026gt; E[\u0026#34;Optimiser step: theta := theta - eta * grad\u0026#34;] E --\u0026gt; F{\u0026#34;More steps?\u0026#34;} F -- \u0026#34;yes\u0026#34; --\u0026gt; A F -- \u0026#34;no\u0026#34; --\u0026gt; G[\u0026#34;Frozen parameters theta*\u0026#34;]Reading the diagram. Step 1–2 is the forward pass: the same computation used at inference time, with the loss attached at the end. Steps 3–4 are the part that exists only during training — backpropagation applies the chain rule layer by layer to obtain one gradient per parameter. The branch at the bottom is where practice departs from theory: the batch is a sample, so each step moves the parameters along a noisy estimate of the true gradient, and the learning rate schedule decides how large those moves may be.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-mermaid-flowchart/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eOne training step is a fixed cycle: the model predicts, the predictions are\nscored against the targets, the error is propagated backwards, and the\nparameters move a little in the direction that reduces it. The diagram below\nshows that cycle; the repetition is the whole of training.\u003c/p\u003e","title":"A Transformer Training Step (flowchart)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nAn axiom is an assumption taken as given, and the ones that hold up a language model are Kolmogorov\u0026rsquo;s three: probabilities are non-negative, the whole space has probability one, and disjoint events add. Everything else about token distributions is derived from them rather than assumed.\nStating them for a finite vocabulary is enough here, because the sample space of a language model at one position is exactly the token set.\nAxiom (probability space of a token distribution). Let \\(\\mathcal{V}\\) be a finite vocabulary and \\(p\\) a function from \\(\\mathcal{V}\\) to the real numbers. Then \\(p\\) is a probability distribution when\n$$ p_v \\geq 0 \\quad \\text{for every } v, \\qquad \\sum_{v \\in \\mathcal{V}} p_v = 1 . $$Countable additivity is the third axiom of the general theory; on a finite vocabulary it is automatic, and the two conditions above are the whole content.\nThe axioms are what make the softmax the natural output layer: it is the parameterisation that satisfies the first two conditions by construction, for any real-valued scores.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-axiom-probability/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eAn axiom is an assumption taken as given, and the ones that hold up a language model\nare Kolmogorov\u0026rsquo;s three: probabilities are non-negative, the whole space has probability\none, and disjoint events add. Everything else about token distributions is derived from\nthem rather than assumed.\u003c/p\u003e","title":"Kolmogorov Axioms for a Token Distribution (axiom)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nAn example is where a formula stops being notation, and this one is small enough to check by hand: three scores, one softmax, three probabilities that must sum to one.\nThe numbers below are the ones the companion snippet prints, so the arithmetic here and the output of a few lines of code can be compared directly.\nExample. Take the score vector \\(z = (2.0,\\, 1.0,\\, 0.1)\\). Exponentiating and normalising,\n$$ (e^{2.0},\\, e^{1.0},\\, e^{0.1}) = (7.389,\\, 2.718,\\, 1.105), \\qquad \\sum_i e^{z_i} = 11.212 , $$so the probabilities are\n$$ p = (0.659,\\, 0.242,\\, 0.099), $$which sum to one to the precision shown. The largest score, only twice the smallest, still receives more than six times its probability, because the exponential amplifies ratios rather than differences.\nTwo things become visible at this size that are hard to see in general form: the ordering of the scores is preserved exactly by the softmax, and the gap between the first and second probability is much wider than the gap between the first two scores.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-example-softmax-three/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eAn example is where a formula stops being notation, and this one is small enough to check\nby hand: three scores, one softmax, three probabilities that must sum to one.\u003c/p\u003e\n\u003cp\u003eThe numbers below are the ones the companion snippet prints, so the arithmetic here and\nthe output of a few lines of code can be compared directly.\u003c/p\u003e","title":"Softmax of Three Scores (example)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nReading a bound off a proposition is the standard use of a corollary, and the one worth having here is a ceiling on how large an attention output can be. Since the output is a convex combination of value vectors, its length can never exceed the longest of them.\nThe consequence is a stability statement that does not depend on the queries at all: however sharp the attention distribution becomes, the outputs stay inside a ball whose radius is set by the values.\nCorollary. If the attention weights of row \\(i\\) form a probability distribution \\(A_{i\\cdot}\\) and \\(o_i = \\sum_j A_{ij} v_j\\), then for any norm\n$$ \\lVert o_i \\rVert \\leq \\max_j \\lVert v_j \\rVert , $$with equality when the row is concentrated on a value vector of maximal norm.\nThe proof is the triangle inequality together with \\(\\sum_j A_{ij} = 1\\): \\(\\lVert \\sum_j A_{ij} v_j \\rVert \\leq \\sum_j A_{ij} \\lVert v_j \\rVert \\leq \\max_j \\lVert v_j \\rVert\\).\nA bound of this kind is what licenses the common practice of normalising the value projection rather than the output: controlling the radius of the value vectors controls the radius of everything downstream.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-corollary-attention-norm/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eReading a bound off a proposition is the standard use of a corollary, and the one worth\nhaving here is a ceiling on how large an attention output can be. Since the output is a\nconvex combination of value vectors, its length can never exceed the longest of them.\u003c/p\u003e","title":"A Bound on the Attention Output Norm (corollary)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nA training run has a shape in time as well as in the loss curve: data preparation that nothing else can start without, a warmup whose length is set by the step size, a long stable phase, and a cooldown whose purpose is to end at a point the optimiser has settled into.\nThis note uses a Gantt chart, which is the diagram type for anything where the order and duration of phases is the content. The bars below are illustrative rather than measured, and the durations are the ones a small decoder run would typically use.\ngantt title A training schedule for a small decoder dateFormat YYYY-MM-DD axisFormat %b %d section Data Tokenise and shard :done, data, 2026-09-01, 3d section Training Warmup :done, warm, after data, 2d Stable phase :active, stable, after warm, 9d Cooldown : cool, after stable, 3d section Evaluation Checkpoint sweeps : eval, after cool, 2dThe dependency arrows matter more than the bar lengths: the schedule encodes the constraint that evaluation cannot begin before a checkpoint exists, and the warmup cannot be skipped because the step size at initialisation is larger relative to the curvature than it is later in the run.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-mermaid-gantt-training/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eA training run has a shape in time as well as in the loss curve: data preparation\nthat nothing else can start without, a warmup whose length is set by the step\nsize, a long stable phase, and a cooldown whose purpose is to end at a point the\noptimiser has settled into.\u003c/p\u003e","title":"A Training Schedule (gantt)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nThe number a language model reports while training is the mean negative log-likelihood of the token that actually came next: softmax cross-entropy over the vocabulary at every position. Its backward pass has an unusually clean closed form — the softmax Jacobian and the logarithm cancel, leaving the probabilities minus the one-hot target, divided by the number of predictions.\nThe code below is the full training-side loop for that quantity: a forward pass that keeps the probabilities its backward will need, the matching backward pass, and a finite-difference check that compares the analytic gradient against central differences one parameter at a time.\nimport numpy as np def softmax_cross_entropy(logits, targets): \u0026#34;\u0026#34;\u0026#34;Forward pass for next-token prediction. logits: (B, T, V) score for every vocabulary entry at each position targets: (B, T) the token that actually came next Returns the mean negative log-likelihood and a cache for the backward. \u0026#34;\u0026#34;\u0026#34; shifted = logits - logits.max(axis=-1, keepdims=True) # stays finite p = np.exp(shifted) p /= p.sum(axis=-1, keepdims=True) # rows sum to 1 B, T, V = logits.shape flat = p.reshape(B * T, V) picked = flat[np.arange(B * T), targets.reshape(-1)] return -np.log(picked).mean(), {\u0026#34;p\u0026#34;: p, \u0026#34;targets\u0026#34;: targets} def softmax_cross_entropy_backward(cache): \u0026#34;\u0026#34;\u0026#34;dL/dlogits = (p - one_hot(target)) / (B * T). The softmax Jacobian and the logarithm cancel, so the gradient of the loss needs no extra matrix product. \u0026#34;\u0026#34;\u0026#34; p, targets = cache[\u0026#34;p\u0026#34;], cache[\u0026#34;targets\u0026#34;] B, T, V = p.shape grad = p.copy() flat = grad.reshape(B * T, V) # a view: grad changes with it flat[np.arange(B * T), targets.reshape(-1)] -= 1.0 return grad / (B * T) def finite_difference(f, x, eps=1e-5): \u0026#34;\u0026#34;\u0026#34;Central-difference gradient, one parameter at a time.\u0026#34;\u0026#34;\u0026#34; grad = np.zeros_like(x) for index in np.ndindex(x.shape): original = x[index] x[index] = original + eps high = f(x) x[index] = original - eps low = f(x) x[index] = original grad[index] = (high - low) / (2 * eps) return grad if __name__ == \u0026#34;__main__\u0026#34;: rng = np.random.default_rng(0) logits = rng.normal(size=(2, 4, 7)) targets = rng.integers(0, 7, size=(2, 4)) loss, cache = softmax_cross_entropy(logits, targets) analytic = softmax_cross_entropy_backward(cache) numeric = finite_difference( lambda x: softmax_cross_entropy(x, targets)[0], logits.copy() ) gap = np.abs(analytic - numeric).max() print(f\u0026#34;loss {loss:.4f}\u0026#34;) print(f\u0026#34;max |analytic - numeric| = {gap:.2e}\u0026#34;) Run as printed, the loss is 2.0409 on random scores — just above \\(\\ln 7 \\approx 1.95\\), the value of a uniform guess over seven tokens — and the two gradients agree to 2.43e-11, so the closed form is correct to the precision of the step size. A check like this is the cheapest way to catch a wrong axis or a missing division before a training run.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-cross-entropy-gradient-python/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eThe number a language model reports while training is the mean negative\nlog-likelihood of the token that actually came next: softmax cross-entropy over\nthe vocabulary at every position. Its backward pass has an unusually clean\nclosed form — the softmax Jacobian and the logarithm cancel, leaving the\nprobabilities minus the one-hot target, divided by the number of predictions.\u003c/p\u003e","title":"Softmax Cross-Entropy and Its Gradient in Python (advanced)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nStep sizes in deep learning are chosen as if the loss were smooth in its parameters, and that is an assumption, not a theorem. It is the smoothness postulate: the gradient does not change arbitrarily fast, so a small step in parameter space produces a predictable change in loss.\nThe assumption is what turns a learning rate from a guess into a bound. Where it fails — across a sharp loss barrier, or at a spike — the same learning rate produces the visible training instabilities that motivate warmup and clipping.\nPostulate (L-smooth loss). Let \\(L(\\theta)\\) be the training loss as a function of the parameters. The smoothness postulate assumes a constant \\(L \\gt 0\\) with\n$$ \\lVert \\nabla L(\\theta) - \\nabla L(\\theta') \\rVert \\leq L \\lVert \\theta - \\theta' \\rVert $$for all parameter vectors, which is equivalent to the loss lying below its tangent parabolas:\n$$ L(\\theta') \\leq L(\\theta) + \\nabla L(\\theta)^{\\top}(\\theta' - \\theta) + \\frac{L}{2} \\lVert \\theta' - \\theta \\rVert^{2}. $$Reading the bound as a function of the step shows the largest safe step for descent is \\(1/L\\).\nNothing about a transformer guarantees such a constant exists globally; the postulate is used locally, inside a trust region where it is a fair description of the loss, which is the same logic that motivates clipping and warmup in the first place.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-postulate-smoothness/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eStep sizes in deep learning are chosen as if the loss were smooth in its parameters, and\nthat is an assumption, not a theorem. It is the smoothness postulate: the gradient does\nnot change arbitrarily fast, so a small step in parameter space produces a predictable\nchange in loss.\u003c/p\u003e","title":"Smoothness of the Loss (postulate)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nA proof is the part of a note that cannot be skipped, and this one is short enough to show the whole pattern: write the definition, factor out the constant, use that it cancels between numerator and denominator.\nThe proof is worth writing out because it is the template for a family of invariance arguments. The same move — factor a common term out of a normalised expression — proves invariance to scale in other normalisations and shows where such invariances stop.\nProof. Let \\(z \\in \\mathbb{R}^{n}\\) and \\(c \\in \\mathbb{R}\\). By definition,\n$$ \\operatorname{softmax}(z + c\\mathbf{1})_i = \\frac{e^{z_i + c}}{\\sum_j e^{z_j + c}} = \\frac{e^{c} e^{z_i}}{e^{c} \\sum_j e^{z_j}} = \\frac{e^{z_i}}{\\sum_j e^{z_j}} = \\operatorname{softmax}(z)_i , $$where the third equality uses \\(e^{z_j + c} = e^{c} e^{z_j}\\) and the factor \\(e^{c}\\), which is positive and finite for every real \\(c\\), cancels. Since this holds for every index \\(i\\), the two vectors are equal.\nTwo remarks on the hypotheses: the argument needs \\(e^{c}\\) to be a common factor, which is why a shift by a constant vector works and a shift by different amounts per coordinate does not; and it needs \\(0 \\lt e^{c} \\lt \\infty\\), which fails in floating point once \\(c\\) is large enough that the exponentials overflow.\nThat last remark is where the lemma stops being a statement about code: mathematically the invariance is exact, while numerically it holds only for the range of \\(c\\) the representation can express, which is the reason the shift is chosen as the negative of the maximum rather than any other constant.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-proof-softmax-shift/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eA proof is the part of a note that cannot be skipped, and this one is short enough to\nshow the whole pattern: write the definition, factor out the constant, use that it cancels\nbetween numerator and denominator.\u003c/p\u003e","title":"Proof of the Softmax Shift Lemma (proof)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nA counterexample is the cheapest way to kill a plausible statement, and the one worth having ready concerns the order of matrix multiplication. The tempting assertion that \\(AB = BA\\) fails for almost every pair of matrices, and it fails on the smallest possible example.\nThe consequence for models is concrete: two linear layers composed in one order compute a different function from the same two layers in the other order, so the order in a transformer block is a modelling decision rather than a coding detail.\nCounterexample. Take\n$$ A = \\begin{pmatrix} 1 \u0026 1 \\\\ 0 \u0026 1 \\end{pmatrix}, \\qquad B = \\begin{pmatrix} 1 \u0026 0 \\\\ 1 \u0026 1 \\end{pmatrix} . $$Then\n$$ AB = \\begin{pmatrix} 2 \u0026 1 \\\\ 1 \u0026 1 \\end{pmatrix}, \\qquad BA = \\begin{pmatrix} 1 \u0026 1 \\\\ 1 \u0026 2 \\end{pmatrix}, $$which are different matrices, so \\(AB \\neq BA\\). Both products are invertible with determinant one, so this is not a degenerate case in any respect.\nWhat is true, and sometimes confused with commutation, is that ranks and traces of products are order-independent when the shapes allow both orders: \\(\\operatorname{tr}(AB) = \\operatorname{tr}(BA)\\) holds even here, with both equal to three.\nThe example also shows why reordering two projections with different roles — queries and keys, say — changes the model rather than merely the notation.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-counterexample-noncommuting/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eA counterexample is the cheapest way to kill a plausible statement, and the one worth\nhaving ready concerns the order of matrix multiplication. The tempting assertion that\n\\(AB = BA\\) fails for almost every pair of matrices, and it fails on the smallest possible\nexample.\u003c/p\u003e","title":"Matrix Products Do Not Commute (counterexample)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nThe log-partition function appears wherever a normalising constant is needed, and it is trapped between two elementary bounds that are enough for most arguments: the largest term below, the largest term plus the log of the number of terms above.\nThe bounds are used for more than estimates. They are the reason the log-partition is Lipschitz with constant one in the maximum norm, which is what makes it safe to compute in log space in the first place.\nLemma (bounds on the log-sum-exp). For \\(z \\in \\mathbb{R}^{n}\\),\n$$ \\max_i z_i \\leq \\log \\sum_{i=1}^{n} e^{z_i} \\leq \\max_i z_i + \\log n . $$The lower bound is the largest term alone; the upper bound follows because every term is at most \\(e^{\\max_i z_i}\\). Both bounds are attained: the lower one when the other terms vanish, the upper one when all terms are equal.\nA gap of at most \\(\\log n\\) is what makes the log-partition usable as a normaliser even when the individual terms differ by hundreds of nats, since the bounds do not depend on that spread.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-lemma-logsumexp/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eThe log-partition function appears wherever a normalising constant is needed, and it is\ntrapped between two elementary bounds that are enough for most arguments: the largest term\nbelow, the largest term plus the log of the number of terms above.\u003c/p\u003e","title":"Log-Sum-Exp Bounds (lemma)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nArchitectures are usually chosen for what they can compute, but the sharper criterion is what they cannot distinguish: matching the symmetries of the problem removes the need to learn them from data. This principle is why convolution appears in vision and why attention appears in sequences.\nIt is a principle rather than a theorem because the symmetry of the problem is a modelling judgement. Once it is identified, however, the consequence is exact: a layer that commutes with the symmetry group reduces the function space being searched rather than merely biasing the search.\nPrinciple (equivariance). Let \\(G\\) be a group acting on inputs and outputs, and let \\(T_g\\) denote the action of \\(g \\in G\\). A layer \\(f\\) is equivariant when\n$$ f(T_g x) = T_g f(x) \\qquad \\text{for every } g \\in G , $$and invariant when \\(f(T_g x) = f(x)\\). The principle is to choose \\(f\\) from the equivariant class when the task itself satisfies \\(f^{\\star}(T_g x) = T_g f^{\\star}(x)\\). Standard instances: translation for a convolution, permutation for a set encoder, rotation for a point-cloud network — and for a transformer with no position embedding, permutation of the tokens.\nThe principle also predicts where an architecture will struggle: a model that is equivariant to a symmetry the task does not have must spend capacity learning to break it, which is why position embeddings are needed at all.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-principle-equivariance/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eArchitectures are usually chosen for what they can compute, but the sharper criterion is\nwhat they cannot distinguish: matching the symmetries of the problem removes the need to\nlearn them from data. This principle is why convolution appears in vision and why attention\nappears in sequences.\u003c/p\u003e","title":"Equivariance as a Design Principle (principle)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nA criterion is a condition that decides whether an argument or a method applies, and this one is the usual substitute for convexity in the analysis of gradient descent: it bounds the loss by the size of its gradient, so a small gradient implies a small excess loss.\nThe criterion is attractive because it is weaker than strong convexity and still sufficient for linear convergence, and because it is stated in a form a training run can be checked against by measuring both quantities.\nCriterion (Polyak-Lojasiewicz). Let \\(L\\) be differentiable with minimum \\(L^{\\star} = \\min_\\theta L(\\theta)\\). The PL criterion holds with constant \\(\\mu \\gt 0\\) when\n$$ \\frac{1}{2} \\lVert \\nabla L(\\theta) \\rVert^{2} \\geq \\mu \\left(L(\\theta) - L^{\\star}\\right) \\qquad \\text{for every } \\theta . $$If \\(L\\) is also \\(L_s\\)-smooth, gradient descent with step \\(\\eta = 1 / L_s\\) satisfies\n$$ L(\\theta_t) - L^{\\star} \\leq \\left(1 - \\frac{\\mu}{L_s}\\right)^{t} \\left(L(\\theta_0) - L^{\\star}\\right), $$so the excess loss decays geometrically and the ratio \\(\\mu / L_s\\) sets the rate.\nThe criterion is a condition to be checked rather than assumed: both \\(L(\\theta) - L^{\\star}\\) and \\(\\lVert \\nabla L(\\theta) \\rVert\\) are measurable during training, so the inequality can be tested on a run and its constant estimated from the data.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-criterion-pl-condition/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eA criterion is a condition that decides whether an argument or a method applies, and this\none is the usual substitute for convexity in the analysis of gradient descent: it bounds the\nloss by the size of its gradient, so a small gradient implies a small excess loss.\u003c/p\u003e","title":"The Polyak-Lojasiewicz Criterion (criterion)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nA corollary is what falls out of a theorem with one substitution, and the useful one here is a floor on perplexity. Since cross-entropy cannot fall below the entropy of the data, and entropy is non-negative, perplexity cannot fall below one.\nThe bound is what makes a reported perplexity of 0.9 a bug report rather than a result: no distribution can assign more probability than certainty to every observed token.\nCorollary. For any model distribution \\(q\\) and any data distribution \\(p\\) with finite support,\n$$ \\operatorname{PPL}(q) = \\exp H(p, q) \\geq \\exp H(p) \\geq 1 , $$with equality throughout if and only if \\(p\\) is concentrated on a single outcome and \\(q = p\\) there.\nThe first inequality is Gibbs\u0026rsquo; inequality composed with the exponential, which is increasing; the second is \\(H(p) \\geq 0\\), which holds because each term \\(-p_v \\log p_v\\) is non-negative.\nIn practice the floor is far above one: natural text has entropy in the low nats per token, so the gap between a model\u0026rsquo;s perplexity and one is not the gap worth tracking — the gap to the entropy of the data is.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-corollary-perplexity-floor/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eA corollary is what falls out of a theorem with one substitution, and the useful one\nhere is a floor on perplexity. Since cross-entropy cannot fall below the entropy of the\ndata, and entropy is non-negative, perplexity cannot fall below one.\u003c/p\u003e","title":"Perplexity Is at Least One (corollary)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nA principle is a guide to design rather than a result about a particular model, and this one is the oldest in statistics: expected error splits into a term from the model being too simple, a term from it being too sensitive to the sample, and a term that no model can remove.\nRead as a design principle, it says that capacity and regularisation are two ends of one dial, and that a change which reduces one term usually pays in the other. That reading is what makes early stopping, weight decay and dataset size comparable interventions.\nPrinciple (bias-variance decomposition). For an estimator \\(\\hat{f}\\) of a target \\(f\\) at an input \\(x\\), with expectation taken over training samples,\n$$ \\mathbb{E}\\!\\left[\\left(\\hat{f}(x) - f(x)\\right)^{2}\\right] = \\underbrace{\\left(\\mathbb{E}[\\hat{f}(x)] - f(x)\\right)^{2}}_{\\text{bias}^{2}} + \\underbrace{\\operatorname{Var}\\!\\left(\\hat{f}(x)\\right)}_{\\text{variance}} + \\underbrace{\\sigma^{2}}_{\\text{noise}} . $$The principle is the reading of the identity: increasing capacity tends to lower the bias term and raise the variance term, and the noise term is fixed by the data.\nIn a large model the two terms are usually discussed under different names — underfitting and overfitting — and the practical consequence of the decomposition is that neither can be diagnosed from the training loss alone, since the variance term is only visible on data the model has not seen.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-principle-bias-variance/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eA principle is a guide to design rather than a result about a particular model, and this\none is the oldest in statistics: expected error splits into a term from the model being too\nsimple, a term from it being too sensitive to the sample, and a term that no model can\nremove.\u003c/p\u003e","title":"The Bias-Variance Trade-Off (principle)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nA formula is a rule written in symbols so that it can be evaluated, and the one nearly every model is trained with is Adam\u0026rsquo;s update: two running averages of the gradient, a correction for the fact that both start at zero, and a step whose size is set by their ratio.\nThe bias-correction step is what the formula is usually remembered for. Without it the first few steps are far too large, because a running average that began at zero underestimates the quantity it is meant to summarise.\nFormula (Adam). With parameters \\(\\theta\\), gradient \\(g_t\\) at step \\(t\\), step size \\(\\eta\\), decay rates \\(\\beta_1, \\beta_2 \\in [0, 1)\\) and a small constant \\(\\varepsilon\\),\n$$ m_t = \\beta_1 m_{t-1} + (1 - \\beta_1) g_t, \\qquad v_t = \\beta_2 v_{t-1} + (1 - \\beta_2) g_t^{2}, $$$$ \\hat{m}_t = \\frac{m_t}{1 - \\beta_1^{t}}, \\qquad \\hat{v}_t = \\frac{v_t}{1 - \\beta_2^{t}}, \\qquad \\theta_t = \\theta_{t-1} - \\eta \\, \\frac{\\hat{m}_t}{\\sqrt{\\hat{v}_t} + \\varepsilon} . $$The squared gradient in \\(v_t\\) is taken element-wise, so the division is also element-wise: each parameter receives a step inversely proportional to the recent size of its own gradient. The defaults \\(\\beta_1 = 0.9\\), \\(\\beta_2 = 0.999\\), \\(\\varepsilon = 10^{-8}\\) come from the original paper and are still the common setting.\nThe division by \\(\\sqrt{\\hat{v}_t}\\) is what makes the method scale-free: multiplying the loss by a constant leaves the update nearly unchanged, which is why the same step size works across architectures that produce gradients of very different magnitudes.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-formula-adam/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eA formula is a rule written in symbols so that it can be evaluated, and the one nearly\nevery model is trained with is Adam\u0026rsquo;s update: two running averages of the gradient, a\ncorrection for the fact that both start at zero, and a step whose size is set by their\nratio.\u003c/p\u003e","title":"The Adam Update (formula)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nEntropy is a convenient scalar summary of an attention row, and the observation reported here is that its average falls as one moves up the layers: early heads spread their weight widely, later heads concentrate it.\nLike most observations about trained networks, this is a tendency rather than a rule, and the note states the measured form and the conditions under which it was measured. The useful part is the consequence for pruning and for interpretability, both of which care whether a head is diffuse or focused.\nObservation. For a trained decoder with \\(L\\) layers and \\(H\\) heads per layer, let \\(A^{(\\ell, h)}\\) be the attention matrix at layer \\(\\ell\\) and head \\(h\\), averaged over a held-out set of sequences. With the row entropy\n$$ \\mathcal{H}^{(\\ell, h)} = -\\frac{1}{T} \\sum_{i=1}^{T} \\sum_{j=1}^{T} A^{(\\ell, h)}_{ij} \\log A^{(\\ell, h)}_{ij} , $$the observation is a decreasing trend in \\(\\mathcal{H}^{(\\ell, h)}\\) as \\(\\ell\\) increases, from values close to \\(\\log T\\) in the first layers towards values well below it in the last ones. Individual heads depart from the trend in both directions.\nAverage entropy is not the same as usefulness: a head that attends uniformly has maximum entropy and carries little information, while a head that always attends to the same relative offset can have low entropy and be essential, so the observation is a description of the population rather than a criterion for any single head.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-observation-attention-entropy/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eEntropy is a convenient scalar summary of an attention row, and the observation reported\nhere is that its average falls as one moves up the layers: early heads spread their weight\nwidely, later heads concentrate it.\u003c/p\u003e","title":"Attention Entropy Falls with Depth (observation)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nA lemma is a small result proved on the way to something larger, and this one is proved in two lines yet carries most of the numerical stability of a softmax: adding the same constant to every score changes nothing.\nThe consequence is the implementation that is actually used — subtract the largest score before exponentiating — and the reason no probability in the output can overflow to infinity while another underflows to zero.\nLemma (shift invariance). For any vector \\(z \\in \\mathbb{R}^{n}\\) and any constant \\(c\\),\n$$ \\operatorname{softmax}(z + c\\mathbf{1}) = \\operatorname{softmax}(z), $$where \\(\\mathbf{1}\\) is the all-ones vector.\nWith \\(c = -\\max_i z_i\\), the largest exponent becomes \\(e^{0} = 1\\), so every term of the sum lies in \\((0, 1]\\) and the computation stays in range for scores of any magnitude.\nThe lemma is a statement about the exact function; floating-point arithmetic only approximates it, which is why the shifted form is preferred in code — it is the version whose rounding errors stay small for the range of scores a model actually produces.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-lemma-softmax-shift/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eA lemma is a small result proved on the way to something larger, and this one is proved\nin two lines yet carries most of the numerical stability of a softmax: adding the same\nconstant to every score changes nothing.\u003c/p\u003e","title":"Softmax Is Invariant to a Shift (lemma)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nThe simplest optimiser with a memory is gradient descent with momentum: keep a running velocity, add the gradient to it, and step in the direction of the velocity rather than the gradient. The algorithm is short, and the behaviour it introduces is worth the two extra lines.\nThe effect is an averaging of successive gradients, which cancels the oscillating part of the noise and amplifies the consistent part. Its cost is a second state vector and one more constant to set.\nAlgorithm (heavy-ball momentum). Given a differentiable loss \\(L\\), a step size \\(\\eta \\gt 0\\), a momentum coefficient \\(\\beta \\in [0, 1)\\), an initial parameter vector \\(\\theta_0\\) and zero initial velocity \\(v_0\\), the steps are:\nSample or compute the gradient \\(g_t = \\nabla L(\\theta_{t-1})\\). Update the velocity: \\(v_t = \\beta v_{t-1} + g_t\\). Update the parameters: \\(\\theta_t = \\theta_{t-1} - \\eta v_t\\). Unrolling the recursion shows that the update is a weighted sum of past gradients with geometrically decaying weights,\n$$ v_t = \\sum_{k=1}^{t} \\beta^{\\,t-k} g_k , $$so the effective number of gradients being averaged is \\(1 / (1 - \\beta)\\): \\(\\beta = 0.9\\) averages roughly the last ten steps.\nThe recursion also explains the pathology of a badly chosen coefficient: with \\(\\beta\\) close to one the effective averaging window is long enough for the direction to lag behind a changing loss surface, and the momentum carries the parameters past the minimum it was approaching.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-algorithm-momentum/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eThe simplest optimiser with a memory is gradient descent with momentum: keep a running\nvelocity, add the gradient to it, and step in the direction of the velocity rather than the\ngradient. The algorithm is short, and the behaviour it introduces is worth the two extra\nlines.\u003c/p\u003e","title":"Gradient Descent with Momentum (algorithm)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nAttention is often drawn as a lookup, and the proposition below says what kind of lookup it is: a convex combination of the value vectors, with weights that are non-negative and sum to one because they came out of a softmax.\nThe statement constrains the output geometry before any training happens, and it is the reason an attention output cannot be extrapolated outside the region its value vectors span — no matter how extreme the queries become.\nProposition. Let \\(A \\in \\mathbb{R}^{T \\times T}\\) be the row-stochastic matrix produced by the softmax in attention, so that \\(A_{ij} \\geq 0\\) and \\(\\sum_j A_{ij} = 1\\) for every row \\(i\\). Then each output row \\(o_i = \\sum_j A_{ij} v_j\\) lies in the convex hull of the value vectors \\(v_1, \\dots, v_T\\).\nA one-line consequence is a bound on the output,\n$$ \\lVert o_i \\rVert_2 \\leq \\max_j \\lVert v_j \\rVert_2 , $$by the triangle inequality and \\(\\sum_j A_{ij} = 1\\).\nThe bound is inherited rather than learned: making attention more confident moves the output towards a vertex of the hull, and no amount of training can move it outside.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-proposition-attention-convex/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eAttention is often drawn as a lookup, and the proposition below says what kind of lookup\nit is: a convex combination of the value vectors, with weights that are non-negative and\nsum to one because they came out of a softmax.\u003c/p\u003e","title":"Attention Outputs Are Convex Combinations (proposition)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nThe same generation loop can be described without messages, as a machine with a small number of states. This view is useful when the interesting questions are about transitions — when decoding starts, what keeps it running, and which conditions end it. The state diagram below is the whole lifecycle of one completion.\nTwo conditions drive the transitions. The first is the probability the model assigns to the end-of-sequence token: decoding stops when it becomes the sampled token, \\(x_t = \\langle \\text{eos} \\rangle\\). The second is a hard budget — a maximum number of tokens — which stops generation that has not converged, and is the reason a truncated answer can look unfinished.\nstateDiagram-v2 [*] --\u0026gt; Idle Idle --\u0026gt; Prefill: prompt received Prefill --\u0026gt; Decode: caches built Decode --\u0026gt; Decode: next token Decode --\u0026gt; Complete: EOS sampled Decode --\u0026gt; Truncated: budget spent Complete --\u0026gt; [*] Truncated --\u0026gt; [*]Reading the diagram. Prefill and Decode are separate states because they cost differently: prefill processes the prompt in parallel, decode advances one position at a time and is bound by memory traffic rather than arithmetic. The self-loop on Decode is where almost all the time is spent. Note also that the two exit states are not equivalent — Complete means the model chose to stop, Truncated means something else did, and downstream code that treats them as the same will occasionally present a cut-off answer as a finished one.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-mermaid-state/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eThe same generation loop can be described without messages, as a machine with\na small number of states. This view is useful when the interesting questions are\nabout \u003cem\u003etransitions\u003c/em\u003e — when decoding starts, what keeps it running, and which\nconditions end it. The state diagram below is the whole lifecycle of one\ncompletion.\u003c/p\u003e","title":"Decoding as a State Machine (state diagram)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nOnce a model trains, the folder structure has a second job: it has to separate what changes per run from what changes per idea. Configurations and checkpoints belong to a run; attention code, data preparation and the training loop belong to the project, and the split is what lets a result be reproduced from a config file plus a commit rather than from memory.\nThis tree is three levels deep and gives each responsibility its own folder: model code, data preparation next to its shards, the entry points under scripts/, and gradient checks under tests/. The two generated folders are listed without their contents, since their files are produced by running the project rather than written into it.\ntiny-transformer/ ├── configs/ │ ├── base.toml # 6 layers, 8 heads, 2k context │ └── smoke.toml # 2 layers, for a one-minute run ├── data/ │ ├── prepare.py # tokenise the corpus and shard it │ └── shards/ │ ├── train.bin │ └── val.bin ├── model/ │ ├── attention.py # scaled dot-product, causal mask │ ├── blocks.py # residual stream, layer norm, MLP │ └── loss.py # softmax cross-entropy ├── scripts/ │ ├── train.py # reads a config, writes a checkpoint │ ├── eval.py # validation loss for one checkpoint │ └── sample.py # top-p decoding from a checkpoint ├── checkpoints/ # written by scripts/train.py ├── tests/ │ └── test_gradients.py # finite-difference checks per module └── README.md The layout pays off at evaluation time, which is where most projects lose their history: a folder of checkpoints plus one evaluation script regenerates the same number months later, and the config beside each checkpoint says which model produced it. The next size up is what appears when a project holds several experiments at once — sweeps, benchmarks and figures — and needs each of them to stay traceable.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-folder-structure-intermediate/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eOnce a model trains, the folder structure has a second job: it has to separate\nwhat changes per run from what changes per idea. Configurations and checkpoints\nbelong to a run; attention code, data preparation and the training loop belong\nto the project, and the split is what lets a result be reproduced from a config\nfile plus a commit rather than from memory.\u003c/p\u003e","title":"Layout of a Trainable Model (intermediate folder structure)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nThe statement that attention outputs are convex combinations follows from two facts about the softmax and one line of the triangle inequality, and the proof shows exactly where each is used.\nWriting it out also shows what would break it. Remove the normalisation of the weights, or allow them to be negative, and the argument fails at the first step — the outputs are then free to leave the hull, which is what happens in attention variants that drop the softmax.\nProof. Let \\(A\\) be the row-stochastic matrix of attention weights, so \\(A_{ij} \\geq 0\\) and \\(\\sum_{j} A_{ij} = 1\\) for every \\(i\\), and let \\(o_i = \\sum_j A_{ij} v_j\\).\nStep one, the weights form a distribution: both properties follow from the definition of the softmax together with the fact that exponentials are strictly positive.\nStep two, the output is a convex combination: a sum of the form \\(\\sum_j A_{ij} v_j\\) with non-negative coefficients adding to one is the definition of a convex combination, so \\(o_i\\) lies in the convex hull of the value vectors by definition of the hull.\nStep three, the norm bound: by the triangle inequality and \\(\\sum_j A_{ij} = 1\\),\n$$ \\lVert o_i \\rVert = \\Bigl\\lVert \\sum_j A_{ij} v_j \\Bigr\\rVert \\leq \\sum_j A_{ij} \\lVert v_j \\rVert \\leq \\Bigl(\\max_j \\lVert v_j \\rVert\\Bigr) \\sum_j A_{ij} = \\max_j \\lVert v_j \\rVert . $$Equality in the last step requires a single non-zero weight on a maximal vector, which is the limit of a fully peaked attention row.\nThe proof is short because the softmax does the work up front: any normalisation that produces non-negative weights summing to one gives the same conclusion, which is why the hull argument is quoted for attention without inspecting the score function at all.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-proof-attention-convex/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eThe statement that attention outputs are convex combinations follows from two facts\nabout the softmax and one line of the triangle inequality, and the proof shows exactly\nwhere each is used.\u003c/p\u003e","title":"Proof: Attention Outputs Lie in the Convex Hull (proof)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nThe variants of attention that appear in long-context work are easier to keep apart by what they replace than by their names: each one gives up the full score matrix and puts something else in its place, and the something else is what the family is called.\nThis note uses a mind map, which suits a taxonomy with no order to it: the four families below are siblings, and the leaves under each are the mechanism rather than a sequence of steps.\nmindmap root((Attention)) Full[Full attention] Dense[scores over every earlier position] Cost[cost grows with T squared] Sparse[Sparse patterns] Window[sliding window of the last w positions] Blocks[fixed pattern of blocks] Linear[Linear attention] Kernel[feature map replaces the softmax] Recurrent[running summary instead of a cache] State[State space] Fixed[state of fixed size] Decay[decay stands in for position]Read this way, the families differ in what they keep: a sparse pattern keeps the softmax and drops most of its entries, a linear method keeps all the entries and drops the softmax, and a state-space model drops the pairwise structure altogether in favour of a recurrence.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-mermaid-mindmap-attention/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eThe variants of attention that appear in long-context work are easier to keep\napart by what they replace than by their names: each one gives up the full score\nmatrix and puts something else in its place, and the something else is what the\nfamily is called.\u003c/p\u003e","title":"Families of Attention (mindmap)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nPositive definiteness is the condition that decides whether a quadratic form curves upwards, and Sylvester\u0026rsquo;s criterion turns it into a finite computation: look at the determinants of the leading blocks of the matrix, from the top left outwards.\nThe criterion is used wherever a second-order quantity has to be checked — a Hessian for a local minimum, a covariance for a Gaussian density, a preconditioner for a second-order optimiser — and each of those settings inherits the same test.\nCriterion (Sylvester). A symmetric matrix \\(M \\in \\mathbb{R}^{n \\times n}\\) is positive definite if and only if every leading principal minor is positive:\n$$ \\det M_k \\gt 0, \\qquad M_k = \\left(M_{ij}\\right)_{1 \\leq i, j \\leq k}, \\qquad k = 1, \\dots, n . $$Two checks worth keeping alongside it: a positive definite matrix has all eigenvalues positive and hence a non-zero determinant, so the criterion is consistent with the spectral definition; and replacing \\(\\gt\\) by \\(\\geq\\) gives the test for positive semi-definiteness, where the minors need not all be positive, which is why the two cases are stated separately.\nFor a machine-learning problem the criterion is mostly used to diagnose failure: a Hessian whose minors change sign marks a saddle rather than a minimum, which is what makes the distinction visible in a computed matrix even when the loss curve is flat.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-criterion-sylvester/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003ePositive definiteness is the condition that decides whether a quadratic form curves\nupwards, and Sylvester\u0026rsquo;s criterion turns it into a finite computation: look at the\ndeterminants of the leading blocks of the matrix, from the top left outwards.\u003c/p\u003e","title":"Sylvester's Criterion for Positive Definiteness (criterion)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nA proposition is a proved statement of moderate weight; this one is the reason the training loss needs no special treatment in the backward pass. Composing softmax with negative log-likelihood, the Jacobian of the softmax and the derivative of the logarithm cancel almost completely.\nWhat survives is an expression that can be read in one line: the gradient with respect to the scores is the predicted distribution minus the observed one.\nProposition. Let \\(p = \\operatorname{softmax}(z)\\) for a score vector \\(z \\in \\mathbb{R}^{V}\\), and let the loss be the negative log-likelihood of a target token \\(y\\), so that \\(L = -\\log p_y\\). Then\n$$ \\frac{\\partial L}{\\partial z} = p - e_y , $$where \\(e_y\\) is the one-hot vector of the target. For a batch of \\(N\\) positions, the mean loss has gradient \\((p - e_y) / N\\) at each of them.\nThe expression is what makes the backward pass through a language-model head cheap, since the gradient at every position is already available as the forward pass output, and it is the quantity a finite-difference check is normally used to verify.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-proposition-gradient-cross-entropy/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eA proposition is a proved statement of moderate weight; this one is the reason the\ntraining loss needs no special treatment in the backward pass. Composing softmax with\nnegative log-likelihood, the Jacobian of the softmax and the derivative of the logarithm\ncancel almost completely.\u003c/p\u003e","title":"The Gradient of Cross-Entropy Through Softmax (proposition)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nThe exponential form of a language model\u0026rsquo;s output is not a design choice made for convenience; it is forced by an optimisation over distributions. Maximise entropy subject to matching a set of expected features, and the solution is always of the form \\(\\exp(\\theta^{\\top} f(x))\\), normalised.\nFor a token distribution constrained only by the scores, that form is the softmax. The theorem is the reason the softmax keeps reappearing in places that look unrelated to classification.\nTheorem (maximum entropy). Among distributions \\(q\\) on a finite set that satisfy \\(\\mathbb{E}_q[f] = \\bar{f}\\) for a given feature map \\(f\\), the one maximising the entropy \\(H(q) = -\\sum_v q_v \\log q_v\\) has the form\n$$ q_v = \\frac{\\exp\\!\\left(\\theta^{\\top} f(v)\\right)}{Z(\\theta)}, \\qquad Z(\\theta) = \\sum_{v} \\exp\\!\\left(\\theta^{\\top} f(v)\\right), $$for some parameter vector \\(\\theta\\) chosen so the constraints hold. With \\(f(v) = z_v\\) and \\(\\theta = 1\\), this is exactly the softmax of the scores.\nRead backwards, the theorem says what a softmax commits to: it is the smoothest distribution consistent with the scores, so any structure beyond them — sparsity, hard constraints — has to be imposed deliberately rather than falling out of the parameterisation.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-theorem-max-entropy/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eThe exponential form of a language model\u0026rsquo;s output is not a design choice made for\nconvenience; it is forced by an optimisation over distributions. Maximise entropy subject\nto matching a set of expected features, and the solution is always of the form\n\\(\\exp(\\theta^{\\top} f(x))\\), normalised.\u003c/p\u003e","title":"Maximum Entropy Gives the Softmax (theorem)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nDefinitions in this subject fix notation rather than prove anything, and attention is the clearest case: three projections, one score matrix, one weighted average. Every implementation, diagram and proof in the rest of the literature is written against this one paragraph of notation.\nThe definition below is stated for a single sequence without the batch dimension. The softmax is taken over the keys, so each row of the score matrix becomes a distribution over the positions a token may read from.\nDefinition (scaled dot-product attention). Let \\(Q, K, V\\) be matrices of shape \\(T \\times d_h\\), called queries, keys and values. The attention output is\n$$ \\operatorname{softmax}\\!\\left(\\frac{QK^{\\top}}{\\sqrt{d_h}}\\right) V , $$where the softmax is applied to every row of the \\(T \\times T\\) score matrix \\(QK^{\\top} / \\sqrt{d_h}\\), and \\(\\sqrt{d_h}\\) is a fixed rescaling that keeps the scores of comparable size as the head width grows.\nTwo details in the definition carry most of the consequences: the softmax is over the keys rather than the values, which makes the output a weighted average of value vectors, and the \\(\\sqrt{d_h}\\) factor makes the score variance independent of the head width under the usual initialisation.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-definition-attention/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eDefinitions in this subject fix notation rather than prove anything, and attention is\nthe clearest case: three projections, one score matrix, one weighted average. Every\nimplementation, diagram and proof in the rest of the literature is written against this\none paragraph of notation.\u003c/p\u003e","title":"Scaled Dot-Product Attention (definition)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nThe training objective of a language model has a name and a formula, and both matter when the loss curve is read: cross-entropy is the expected negative log-probability of what actually happened, measured in nats.\nThe definition is written here for a single position, then specialised to the case a language model is always in — a target distribution concentrated on one token.\nDefinition (cross-entropy). For two probability distributions \\(p\\) and \\(q\\) on the same finite set, the cross-entropy of \\(q\\) relative to \\(p\\) is\n$$ H(p, q) = -\\sum_{v} p_v \\log q_v , $$with the convention \\(0 \\log 0 = 0\\). When \\(p\\) is the one-hot distribution of the token that actually came next, the sum collapses to a single term,\n$$ H(p, q) = -\\log q_{y}, $$which is the quantity minimised at every position of a training sequence.\nWriting the one-hot case separately is what makes the loss comparable across vocabularies of different sizes: the generic form says what the objective is, and the collapsed form says what is actually computed, per token, per position.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-definition-cross-entropy/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eThe training objective of a language model has a name and a formula, and both matter\nwhen the loss curve is read: cross-entropy is the expected negative log-probability of\nwhat actually happened, measured in nats.\u003c/p\u003e","title":"Cross-Entropy (definition)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nThe matrix of pairwise inner products of a set of vectors has a property that constrains what similarity patterns are possible: it is symmetric and positive semi-definite, so its eigenvalues are non-negative and its diagonal entries are the squared lengths of the vectors.\nThe property is used in both directions. Forward, it says that a kernel matrix computed from a feature map is automatically a valid Gram matrix; backward, it says that an arbitrary table of similarities is not necessarily realisable by any embedding.\nProperty. For vectors \\(x_1, \\dots, x_n \\in \\mathbb{R}^{d}\\), the Gram matrix \\(G_{ij} = \\langle x_i, x_j \\rangle\\) satisfies\n$$ G = X X^{\\top} \\succeq 0, \\qquad G_{ii} = \\lVert x_i \\rVert^{2} , $$for the matrix \\(X\\) with rows \\(x_i\\). Positive semi-definiteness follows from \\(u^{\\top} G u = \\lVert X^{\\top} u \\rVert^{2} \\geq 0\\) for every \\(u\\), and it implies that every eigenvalue of \\(G\\) is non-negative and that \\(\\lvert G_{ij} \\rvert \\leq \\sqrt{G_{ii} G_{jj}}\\) by Cauchy-Schwarz.\nThe self-attention score matrix \\(Q Q^{\\top}\\) is an instance: it records the similarity of each query with each other query, and its diagonal is the squared norm of each query.\nBecause the property is a constraint, it also serves as a test: a similarity table whose eigenvalues include a clearly negative one cannot have come from inner products of real vectors, so the feature map that supposedly produced it is mis-specified.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-property-gram-psd/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eThe matrix of pairwise inner products of a set of vectors has a property that constrains\nwhat similarity patterns are possible: it is symmetric and positive semi-definite, so its\neigenvalues are non-negative and its diagonal entries are the squared lengths of the\nvectors.\u003c/p\u003e","title":"The Gram Matrix Is Positive Semi-Definite (property)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nThe largest singular value of a weight matrix is one of the numbers used to judge how a layer transforms its input, and it is usually estimated rather than computed: repeated multiplication by the matrix makes any starting vector align with the leading eigenvector, after which the Rayleigh quotient gives the value.\nThis note uses the loop style of the pseudocode environment — a while with a stopping test and per-line comments. The stopping test compares the residual to the current eigenvalue estimate rather than to a fixed constant, so the same tolerance works at any scale of matrix.\n\\begin{algorithm}[H] \\footnotesize \\caption{Power iteration for the leading eigenvalue} \\begin{algorithmic}[1] \\Require symmetric matrix \\(A\\), start vector \\(v\\), tolerance \\(\\tau\\) \\State \\(v \\gets v / \\lVert v \\rVert\\) \\Comment{any direction will do} \\State \\(w \\gets A v\\); \\(\\lambda \\gets \\lVert w \\rVert\\) \\Comment{first Rayleigh estimate} \\While{\\(\\lVert w - \\lambda v \\rVert \\gt \\tau \\, \\lambda\\)} \\State \\(v \\gets w / \\lambda\\) \\Comment{renormalise, or the vector overflows} \\State \\(w \\gets A v\\); \\(\\lambda \\gets \\lVert w \\rVert\\) \\EndWhile \\Return \\(\\lambda, v\\) \\end{algorithmic} \\end{algorithm} The loop is written without a step limit, which is honest about the mathematics and optimistic in practice: the convergence rate is the ratio of the two largest eigenvalues, so an ill conditioned matrix can need many iterations, and an implementation normally adds a maximum step count to the stopping test for that reason.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-pseudocode-power-iteration/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eThe largest singular value of a weight matrix is one of the numbers used to\njudge how a layer transforms its input, and it is usually estimated rather than\ncomputed: repeated multiplication by the matrix makes any starting vector align\nwith the leading eigenvector, after which the Rayleigh quotient gives the value.\u003c/p\u003e","title":"Power Iteration with a Convergence Check (pseudocode)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nLayer normalisation rescales each token\u0026rsquo;s activations using statistics computed across the features of that token alone, which is what separates it from batch normalisation: nothing in the computation depends on the other examples in the batch.\nThis note uses the function style of the pseudocode environment, with a named procedure, a parameter list and an explicit return. The mathematical content is three lines — a mean, a variance and an affine map — and the interesting part is the placement of the constant \\(\\epsilon\\), which sits inside the square root so that a zero-variance input cannot divide by zero.\n\\begin{algorithm}[H] \\footnotesize \\caption{Layer normalisation of one token\u0026#39;s activations} \\begin{algorithmic}[1] \\Function{LayerNorm}{\\(x, \\gamma, \\beta, \\epsilon\\)} \\State \\(\\mu \\gets \\frac{1}{d} \\sum_{i=1}^{d} x_i\\) \\State \\(\\sigma^{2} \\gets \\frac{1}{d} \\sum_{i=1}^{d} (x_i - \\mu)^{2}\\) \\State \\(\\hat{x} \\gets (x - \\mu) / \\sqrt{\\sigma^{2} + \\epsilon}\\) \\Return \\(\\gamma \\odot \\hat{x} + \\beta\\) \\EndFunction \\end{algorithmic} \\end{algorithm} Two properties follow from the three lines. The mean and variance of \\(\\hat{x}\\) are \\(0\\) and approximately \\(1\\) by construction, up to the effect of \\(\\epsilon\\); and the learned scale and shift mean the layer as a whole can undo the normalisation if that is what the loss prefers, so the operation removes scale information only from the intermediate representation.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-pseudocode-layer-norm/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eLayer normalisation rescales each token\u0026rsquo;s activations using statistics computed\nacross the features of that token alone, which is what separates it from batch\nnormalisation: nothing in the computation depends on the other examples in the\nbatch.\u003c/p\u003e","title":"Layer Normalisation Written as a Function (pseudocode)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nA claim is a statement put forward as true and argued for, not yet established as a theorem. The one here concerns temperature scaling: raising the temperature flattens a distribution, and it cannot make the most probable token more probable than it already was.\nThe claim is worth stating because the opposite is sometimes assumed — that a large temperature might sharpen a peaky distribution. The argument below shows why the largest probability is monotone in the temperature, so no setting of it can increase that maximum.\nClaim. Let \\(z \\in \\mathbb{R}^{V}\\) be a score vector and let \\(p^{(T)} = \\operatorname{softmax}(z / T)\\) for \\(T \\gt 0\\). Writing \\(M(T) = \\max_v p^{(T)}_v\\), the claim is that\n$$ M(T_2) \\leq M(T_1) \\qquad \\text{whenever } T_2 \\geq T_1 \\gt 0 . $$The argument: the temperature enters only through the ratio \\(z / T\\), which shrinks the spread of the scores as \\(T\\) grows, and the softmax is a monotone function of that spread. Equivalently, for \\(T \\geq 1\\) the transformed scores \\(z/T\\) lie in the convex hull of the original scores and the origin, and the maximum probability of a distribution is a convex function of its scores.\nThe claim quantifies a lever rather than a guarantee: temperature cannot create confidence, it can only redistribute the confidence the scores already justify — which is why sampling at high temperature does not make a model more accurate, only more diverse.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-claim-temperature-monotone/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eA claim is a statement put forward as true and argued for, not yet established as a\ntheorem. The one here concerns temperature scaling: raising the temperature flattens a\ndistribution, and it cannot make the most probable token more probable than it already\nwas.\u003c/p\u003e","title":"Temperature Cannot Sharpen (claim)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nConvexity of a loss in the parameters is rare in deep learning, but convexity in the scores is not, and the cross-entropy of a softmax has it. The property is what guarantees that the inner optimisation — fitting the output layer, probing a frozen representation — has no local minima to fall into.\nThe property is stated with respect to the logits, not the weights. As soon as the logits depend non-linearly on parameters, the composition need not be convex, which is exactly the boundary the statement respects.\nProperty. For a fixed target distribution \\(p\\), the function\n$$ z \\mapsto H(p, \\operatorname{softmax}(z)) = -\\sum_v p_v \\log \\operatorname{softmax}(z)_v $$is convex on \\(\\mathbb{R}^{V}\\), and strictly convex on the affine subspace where the softmax is not constant. The derivative with respect to \\(z\\),\n$$ \\nabla_z H = \\operatorname{softmax}(z) - p , $$vanishes only where the predicted distribution equals the target, which is therefore the unique minimiser up to the shift redundancy of the softmax.\nThe property is why logistic regression is a convex problem and why fitting a linear probe on frozen features is reliable: in both cases the only non-convex part of the computation has been frozen, leaving an objective with a single basin.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-property-cross-entropy-convex/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eConvexity of a loss in the parameters is rare in deep learning, but convexity in the\nscores is not, and the cross-entropy of a softmax has it. The property is what guarantees\nthat the inner optimisation — fitting the output layer, probing a frozen representation —\nhas no local minima to fall into.\u003c/p\u003e","title":"Cross-Entropy Is Convex in the Logits (property)"},{"content":"DyCo Skills in one line: humans choose and oversee, agents draft and scale, the workflow is dynamic.\nflowchart TD A[Start] --\u0026gt; B{Is the chart wide?} B -- yes --\u0026gt; C[Scroll horizontally] B -- no --\u0026gt; D[Fit the column] C --\u0026gt; E[End] D --\u0026gt; E","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-with-mermaid/","summary":"\u003cp\u003eDyCo Skills in one line: humans choose and oversee, agents draft and scale, the workflow is dynamic.\u003c/p\u003e\n\u003cpre class=\"mermaid\"\u003eflowchart TD\n    A[Start] --\u0026gt; B{Is the chart wide?}\n    B -- yes --\u0026gt; C[Scroll horizontally]\n    B -- no --\u0026gt; D[Fit the column]\n    C --\u0026gt; E[End]\n    D --\u0026gt; E\u003c/pre\u003e","title":"Note with Mermaid"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nGradient accumulation is the standard way to train with a batch larger than memory allows: several small batches are processed in turn and their gradients added before any parameter moves. The gradient of a sum is the sum of the gradients, so the result equals one step on the concatenated batch as long as the per-batch losses are averaged.\nThis note writes the procedure in the contract style, stating what it requires and what it guarantees before any step is listed. Diagrams of the same loop usually omit the division by the number of micro-batches, which is the one detail that makes the accumulated gradient comparable to a single large-batch gradient.\n\\begin{algorithm}[H] \\footnotesize \\caption{Gradient accumulation over micro-batches} \\begin{algorithmic}[1] \\Require parameters \\(\\theta\\), micro-batches \\(B_1, \\dots, B_k\\), learning rate \\(\\eta\\) \\Ensure parameters after one accumulated step \\State \\(g \\gets 0\\) \\For{\\(i = 1, \\dots, k\\)} \\State evaluate the loss \\(L_i\\) on \\(B_i\\) with the current \\(\\theta\\) \\State \\(g \\gets g + \\nabla_\\theta L_i / k\\) \\EndFor \\State \\(\\theta \\gets \\theta - \\eta g\\) \\Return \\(\\theta\\) \\end{algorithmic} \\end{algorithm} The contract is what makes the procedure checkable: the number of backward passes equals the number of micro-batches, the parameters move exactly once, and the accumulated gradient matches the mean gradient over the whole set of micro-batches to within the arithmetic of the summation order.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-pseudocode-gradient-accumulation/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eGradient accumulation is the standard way to train with a batch larger than\nmemory allows: several small batches are processed in turn and their gradients\nadded before any parameter moves. The gradient of a sum is the sum of the\ngradients, so the result equals one step on the concatenated batch as long as the\nper-batch losses are averaged.\u003c/p\u003e","title":"Gradient Accumulation with an Input and Output Contract (pseudocode)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nA language model ends in a softmax: the raw scores of the last layer become a probability distribution over the vocabulary, and everything downstream — the training loss, sampling, beam search — reads that distribution rather than the scores themselves.\nThe snippet below is the smallest useful piece of that head: one function, one short array, and two printed lines. The line to notice is the shift before the exponential — subtracting the largest score leaves the distribution unchanged but keeps exp in range, so a score of 800 gives the same answer as a score of 0 instead of overflowing to infinity.\nimport numpy as np def softmax(logits): \u0026#34;\u0026#34;\u0026#34;Turn a row of logits into a probability distribution.\u0026#34;\u0026#34;\u0026#34; shifted = logits - logits.max(axis=-1, keepdims=True) # stays finite e = np.exp(shifted) return e / e.sum(axis=-1, keepdims=True) logits = np.array([2.0, 1.0, 0.1]) p = softmax(logits) print(\u0026#34;probabilities\u0026#34;, np.round(p, 3)) print(\u0026#34;sum\u0026#34;, p.sum()) Run as printed, it reports [0.659 0.242 0.099] and a sum of 1.0. Dividing the scores by a temperature before the softmax sharpens or flattens the same distribution, which is the knob that sampling turns.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-softmax-python/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eA language model ends in a softmax: the raw scores of the last layer become a\nprobability distribution over the vocabulary, and everything downstream — the\ntraining loss, sampling, beam search — reads that distribution rather than the\nscores themselves.\u003c/p\u003e","title":"Numerically Stable Softmax in Python (starter)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nLong-context attention is usually chosen on cost rather than on quality. What decides whether a variant is usable at a given sequence length is how the score cost and the per-token cache grow with \\(T\\), and both are easiest to compare side by side in one table rather than across several paragraphs.\nThe table below uses the striped style ({.table-striped}): horizontal rules and a tinted every-other row, which keeps a row readable once it runs past five or six columns. Its cells hold plain text, typeset math, and a line break where two short answers belong in one cell, so a wide comparison still fits one screen.\nVariant What it attends to Score cost Cache per token Position signal Parallel over \\(T\\) Full every earlier position \\(T^{2}\\) \\(2 \\cdot d_{\\text{model}}\\) rotary or additive yes Sliding window the last \\(w\\) positions \\(T w\\) \\(2 \\cdot d_{\\text{model}}\\) rotary yes Block sparse a fixed pattern of blocks \\(T \\sqrt{T}\\) \\(2 \\cdot d_{\\text{model}}\\) rotary yes Kernel (linear) a running feature summary \\(T d_{\\varphi}\\) \\(2 \\cdot d_{\\varphi}\\) decayed features yes State space a compressed recurrent state \\(T d_{s}\\) \\(d_{s}\\) exponential decay sequential\nscan The cache column is the one that decides serving cost. A full-attention layer keeps \\(2 \\cdot d_{\\text{model}}\\) numbers per token, so its memory grows with sequence length even when the score cost is no longer the bottleneck, while a state-space layer replaces that growth with a state whose size is fixed by design.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-attention-variants-table/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eLong-context attention is usually chosen on cost rather than on quality. What\ndecides whether a variant is usable at a given sequence length is how the score\ncost and the per-token cache grow with \\(T\\), and both are easiest to compare\nside by side in one table rather than across several paragraphs.\u003c/p\u003e","title":"Attention Variants for Long Contexts (striped table)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nA conjecture is a statement believed on the weight of evidence but not proved, and the best-known one in this field concerns how loss falls as models grow. Fits are tight over the measured range, and the conjecture is that the same form continues past it.\nNothing in the mathematics forces the form; it is adopted because the residuals of the fit are small and because the constants that come out are stable across independent studies. That is strong evidence and it is still not a proof.\nConjecture (power-law scaling). There exist constants \\(L_\\infty \\geq 0\\), \\(N_c \\gt 0\\) and \\(\\alpha \\gt 0\\) such that, for models trained to convergence on a fixed token budget,\n$$ L(N) = L_{\\infty} + \\left(\\frac{N_c}{N}\\right)^{\\alpha} $$holds for every model size \\(N\\), and similarly in the number of training tokens \\(D\\) with its own exponent \\(\\beta\\). The conjecture asserts the form outside the range where it has been fitted.\nA consequence used in planning runs: doubling the parameter count reduces the size-dependent term by \\(2^{-\\alpha}\\), so a small difference in \\(\\alpha\\) compounds into a large difference in compute.\nBecause the statement is a conjecture, the honest way to use it is as a budget model: it says what a run is expected to buy, and any run whose measured loss sits well off the fitted curve is evidence about the form itself rather than only about that run.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-conjecture-scaling/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eA conjecture is a statement believed on the weight of evidence but not proved, and the\nbest-known one in this field concerns how loss falls as models grow. Fits are tight over\nthe measured range, and the conjecture is that the same form continues past it.\u003c/p\u003e","title":"Power-Law Scaling (conjecture)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nMulti-head attention reads better as shape bookkeeping than as prose: every symbol below is a tensor, and the shapes alone decide which products are legal, which is why papers about attention variants usually open with a notation table before showing a single result.\nThe table below uses the academic three-rule style ({.table-academic}): a rule above the table, a thin rule under the header, a rule at the bottom, and no vertical lines — the shape journals and conference proceedings ask for. The style is chosen per table with a block attribute on the Markdown table, so a note can mix it with the default style without editing any stylesheet.\nSymbol Shape Meaning \\(X\\) \\(T \\times d_{\\text{model}}\\) input embeddings of a sequence of \\(T\\) tokens \\(W^{Q}, W^{K}, W^{V}\\) \\(d_{\\text{model}} \\times d_{\\text{model}}\\) learned projections producing queries, keys and values \\(Q, K, V\\) \\(T \\times d_{\\text{model}}\\) the projected sequence, one matrix per role \\(A\\) \\(T \\times T\\) attention weights, with every row summing to one \\(H\\) scalar number of heads the projections are split into \\(d_{h}\\) scalar head width, \\(d_{\\text{model}} / H\\) \\(P\\) \\(T \\times d_{\\text{model}}\\) concatenated head outputs, before the final projection Shapes are written without the batch dimension. Adding it gives \\(B \\times T \\times d_{\\text{model}}\\) for the inputs, and the per-head views become \\(B \\times H \\times T \\times d_{h}\\) once each projection is split — the layout most implementations keep in memory, and the one that explains why the head count \\(H\\) costs nothing in total width as long as \\(d_{h}\\) shrinks with it.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-attention-notation-table/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eMulti-head attention reads better as shape bookkeeping than as prose: every\nsymbol below is a tensor, and the shapes alone decide which products are legal,\nwhich is why papers about attention variants usually open with a notation table\nbefore showing a single result.\u003c/p\u003e","title":"Notation for Multi-Head Attention (academic table)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nEvaluating a system that uses tools needs its own body of sources: benchmarks for tool use, for long-context reasoning and for end-to-end knowledge work, together with the model reports those numbers are read against. This note keeps that literature in a bibliography of its own, separate from the prompting and fine-tuning lists used elsewhere on this site.\nTool-use competence is measured by MCP-Atlas \\cite{bandi2026mcpatlas}, long-context reasoning by the AA-LCR benchmark \\cite{aalcr}, and the translation result that most attention papers still compare against is the one in Bahdanau et al. \\cite{bahdanau2014attention}.\nReferences The three entries above come from a file that shares no keys with the other bibliographies here, which is the point of naming a file per list: this note downloads only its own references, and a key that lives in another paper\u0026rsquo;s file would be reported as missing rather than silently resolved.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-third-bibliography-benchmarks/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eEvaluating a system that uses tools needs its own body of sources: benchmarks for\ntool use, for long-context reasoning and for end-to-end knowledge work, together\nwith the model reports those numbers are read against. This note keeps that\nliterature in a bibliography of its own, separate from the prompting and\nfine-tuning lists used elsewhere on this site.\u003c/p\u003e","title":"Benchmarks for Agentic Systems (demo)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nConvexity is often described as the property that guarantees a unique optimum, and the statement is false as written: convex functions can have a whole interval of minimisers. The counterexample is a flat-bottomed function, convex everywhere and constant on a segment.\nThe distinction that survives is between convex and strictly convex, and it matters in practice because a flat region is exactly where a gradient method stalls without having found an isolated point.\nCounterexample. Define \\(f : \\mathbb{R} \\to \\mathbb{R}\\) by\n$$ f(x) = \\max(0,\\, \\lvert x \\rvert - 1) . $$The function is convex, as a maximum of convex functions composed with the convex \\(\\lvert x \\rvert\\), and it is zero on the whole interval \\([-1, 1]\\). Every point of that interval is a global minimiser, so the minimiser set has more than one element; gradient descent started inside the interval sees \\(f'(x) = 0\\) and stops at whichever point it reached first.\nStrict convexity rules this out: if \\(f\\) were strictly convex with a minimiser, that minimiser would be unique.\nFor a trained model the flat region is a shape to look for rather than a pathology: it means many parameter settings are equally good on the objective, and any preference among them has to come from regularisation rather than from the loss.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-counterexample-convex-many-minima/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eConvexity is often described as the property that guarantees a unique optimum, and the\nstatement is false as written: convex functions can have a whole interval of minimisers.\nThe counterexample is a flat-bottomed function, convex everywhere and constant on a\nsegment.\u003c/p\u003e","title":"A Convex Function with Many Minima (counterexample)"},{"content":"Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.\nA language model does not emit a token; it emits a probability for every token in the vocabulary, and something has to choose one. Choosing the single most probable token makes output repetitive, while sampling from the full distribution occasionally picks from its long tail of near-zero probabilities. Nucleus sampling keeps the smallest set of tokens whose combined probability reaches a threshold and samples only from those.\nThe threshold makes the candidate set adaptive rather than fixed: after a peaked distribution, the nucleus is one or two tokens; after a flat one, it can hold hundreds. Temperature acts first, dividing the logits before the softmax, so it reshapes the distribution that the nucleus is then cut from.\n\\begin{algorithm}[H] \\caption{Nucleus (Top-p) Sampling of One Token} \\begin{algorithmic}[1] \\Require Logits \\(z \\in \\mathbb{R}^{V}\\) for the next position \\Require Temperature \\(T \u0026gt; 0\\) and nucleus threshold \\(p \\in (0, 1]\\) \\Require Random source \\(u \\sim \\mathrm{Uniform}(0, 1)\\) \\Ensure Sampled token index \\(i^{\\star}\\) \\State Turn the logits into a distribution: \\(q \\leftarrow \\mathrm{softmax}(z / T)\\) \\State Sort the vocabulary by probability, largest first: \\State \\hspace{0.5cm} \\(q_{(1)} \\geq q_{(2)} \\geq \\ldots \\geq q_{(V)}\\) \\State Find the smallest prefix whose probability mass reaches the threshold: \\State \\hspace{0.5cm} \\(k \\leftarrow \\min \\{ m : \\sum_{j=1}^{m} q_{(j)} \\geq p \\}\\) \\State Renormalise the kept tokens so they sum to one: \\State \\hspace{0.5cm} \\(\\tilde{q}_{(j)} \\leftarrow q_{(j)} / \\sum_{l=1}^{k} q_{(l)}\\) for every \\(j \\leq k\\) \\State Sample by walking the cumulative distribution with \\(u\\): \\State \\hspace{0.5cm} \\(i^{\\star} \\leftarrow \\) the first index where the running sum of \\(\\tilde{q}\\) exceeds \\(u\\) \\Return \\(i^{\\star}\\) \\end{algorithmic} \\end{algorithm} Reading the algorithm. Steps 1–2 produce a sorted distribution; step 3 is the whole idea — the size of the candidate set is a consequence of the model\u0026rsquo;s own confidence, not a constant. Step 4 matters for correctness rather than appearance: after truncation the probabilities no longer sum to one, so sampling without renormalisation would bias the choice towards the first kept token. The random draw in step 5 is why the same prompt can yield different completions, and why reproducibility requires fixing the seed as well as the temperature.\n","permalink":"https://www.zhumengyao.com/scidraft/notes/demo-note-top-p-sampling-pseudocode/","summary":"\u003cp\u003e\u003cstrong\u003eDemo note.\u003c/strong\u003e This note demonstrates what the \u003cstrong\u003eSciDraft\u003c/strong\u003e Hugo theme can do\nfor academic and scientific publishing — for research institutions, researchers\nand students. Its content is demonstration material only: readers should ignore\nwhether the demo content is correct.\u003c/p\u003e\n\u003cp\u003eA language model does not emit a token; it emits a probability for every token\nin the vocabulary, and something has to choose one. Choosing the single most\nprobable token makes output repetitive, while sampling from the full distribution\noccasionally picks from its long tail of near-zero probabilities. Nucleus\nsampling keeps the smallest set of tokens whose combined probability reaches a\nthreshold and samples only from those.\u003c/p\u003e","title":"Nucleus (Top-p) Sampling"},{"content":"This A–Z page lists every mathematical result covered in the Math note series. Each entry gives the result\u0026rsquo;s name, the family it belongs to, and a link to the note that states or proves it.\nTo browse by result family instead, visit Math Categories.\n37 Results Covered A B C D E F G H I J K L M N O P Q R S T U V W X Y Z A A Bound on the Attention Output Norm Also known as: Attention Norm Bound, Output Norm Corollary, Bounded Attention Output\nReads off a ceiling on the norm of an attention output as a direct consequence of the convex-combination proposition.\nA Convex Function with Many Minima Also known as: Non-Unique Optimum, Convex Counterexample, Flat Minimum Example\nShows by construction that convexity does not guarantee a unique minimiser, contradicting a common shorthand.\nAttention Entropy Falls with Depth Also known as: Attention Entropy Decay, Entropy vs Depth, Attention Concentration\nReports the empirical finding that attention distributions become more concentrated in deeper layers.\nAttention Outputs Are Convex Combinations Also known as: Attention Convexity Proposition, Convex Combination Proposition, Attention Weights Sum to One\nStates that attention weights are non-negative and sum to one, so each output is a convex combination of the values.\nAttention with Two Tokens Also known as: Two-Token Attention Example, Minimal Attention Worked Example, Single-Head Example\nWorks the smallest interesting attention case — two positions, one head — end to end so every number can be checked by hand.\nB Bayes\u0026#39; Rule Also known as: Bayes\u0026#39; Theorem, Bayes\u0026#39; Formula, Posterior Rule\nRelates a posterior to a likelihood and a prior, giving the rule for updating belief in light of evidence.\nC Cross-Entropy Also known as: Cross-Entropy Loss, Negative Log-Likelihood, Log Loss\nDefines the training objective of a language model as the expected negative log-probability assigned to the true tokens.\nCross-Entropy Is Convex in the Logits Also known as: Convexity of Cross-Entropy, Log-Sum-Exp Convexity, Convex Loss Property\nEstablishes that cross-entropy is convex in the logits, which is why the training objective has no spurious local minima.\nE Equivariance as a Design Principle Also known as: Equivariance Principle, Symmetry as Inductive Bias, Group Equivariance\nTreats a symmetry of the problem as a design constraint, so that transforming the input transforms the output in the matching way.\nF Features Are Directions Also known as: Linear Representation Hypothesis, Feature Directions, Concept Vectors\nConjectures that a network\u0026#39;s learned concepts correspond to directions in activation space that can be added and compared linearly.\nG Gradient Descent with Momentum Also known as: Heavy Ball Method, Momentum Optimisation, EMA Gradient Descent\nKeeps a running average of past gradients so that descent accumulates speed along consistent directions and damps oscillation.\nI Inner-Product Axioms Also known as: Inner Product Space Axioms, Similarity Axioms, Positive-Definite Form\nFixes the three conditions a similarity measure must satisfy for attention scores to behave as inner products.\nK Kolmogorov Axioms for a Token Distribution Also known as: Kolmogorov Axioms, Probability Space Axioms, Normalised Non-Negative Measure\nStates the three assumptions — non-negativity, unit total mass, and additivity — that make a token distribution a probability distribution.\nL Log-Sum-Exp Bounds Also known as: LSE Inequalities, Max-Plus Bounds\nBounds the log-sum-exp of a set of values between their maximum and that maximum plus a log-count term.\nM Masked Softmax Attention Also known as: Causal Masking, Masked Attention, Autoregressive Softmax\nSpecifies how a causal mask is applied inside softmax attention so that each position attends only to itself and earlier positions.\nMatrix Products Do Not Commute Also known as: Non-Commutativity of Matrices, AB ≠ BA, Order Dependence\nDemonstrates with two explicit matrices that reversing a product changes the result, so matrix order cannot be assumed interchangeable.\nMaximum Entropy Gives the Softmax Also known as: Maximum Entropy Principle, MaxEnt Derivation of Softmax, Entropy-Maximising Distribution\nDerives the softmax distribution as the unique maximiser of entropy subject to matching expected scores.\nP Perplexity Is at Least One Also known as: Perplexity Lower Bound, Perplexity ≥ 1, Entropy Floor\nFollows from the definition of perplexity that its value can never drop below one, however good the model.\nPower-Law Scaling Also known as: Scaling Laws, Power-Law Scaling Hypothesis, Compute-Optimal Scaling\nHolds that loss falls as a power law in model size, data and compute, so performance can be extrapolated before training.\nProof of the Softmax Shift Lemma Also known as: Softmax Shift Proof, Translation Invariance Proof, Max-Subtraction Correctness\nProves the shift-invariance lemma by factoring the added constant out of numerator and denominator.\nProof: Attention Outputs Lie in the Convex Hull Also known as: Convex Hull Proof, Attention as Convex Combination, Output Hull Membership Proof\nProves that an attention output is a convex combination of its value vectors, hence lies inside their convex hull.\nR Rotary Position Embedding Also known as: RoPE, Rotary Positional Encoding\nEncodes position by rotating query and key vectors, so attention scores depend on relative distance rather than absolute index.\nS Scaled Dot-Product Attention Also known as: Softmax Attention, Query-Key-Value Attention\nFixes the notation for attention: three projections, one score matrix, and a softmax-weighted average over the values.\nSmoothness of the Loss Also known as: Smoothness Assumption, Lipschitz Gradient Assumption, Bounded Curvature\nAssumes the loss has a bounded gradient Lipschitz constant, the condition most convergence proofs are stated under.\nSoftmax Is Invariant to a Shift Also known as: Softmax Shift Invariance, Translation Invariance of Softmax, Max-Subtraction Lemma\nEstablishes that adding a constant to every logit leaves softmax unchanged, which is what makes max-subtraction safe.\nSoftmax of Three Scores Also known as: Softmax Worked Example, Three-Score Softmax, Numerical Softmax Demo\nEvaluates softmax on three scores so the normalisation step is visible as arithmetic rather than notation.\nSylvester\u0026#39;s Criterion for Positive Definiteness Also known as: Sylvester\u0026#39;s Criterion, Leading Principal Minors Test, Positive Definiteness Test\nDecides positive definiteness by checking the signs of a matrix\u0026#39;s leading principal minors rather than inspecting its eigenvalues.\nT Temperature Cannot Sharpen Also known as: Temperature Monotonicity, No Sharpening Claim, Entropy Floor at Any Temperature\nAsserts that lowering the softmax temperature reweights probabilities without ever making the distribution more peaked than its logits allow.\nThe Adam Update Also known as: Adam Optimiser, Adaptive Moment Estimation, Adam Update Rule\nWrites the Adam update in symbols, combining a momentum term with a running second-moment estimate and a bias correction.\nThe Bias-Variance Trade-Off Also known as: Bias-Variance Trade-Off, Bias-Variance Decomposition, Capacity Trade-Off\nStates that expected error decomposes into bias and variance, so added capacity trades one against the other.\nThe Chain Rule Also known as: Chain Rule, Composite Differentiation Rule, Chain Rule of Calculus\nStates that the derivative of a composition is the product of the derivatives of its parts, the basis of backpropagation.\nThe Gradient of Cross-Entropy Through Softmax Also known as: Softmax Gradient Proposition, Cross-Entropy Gradient, Softmax-Cross-Entropy Derivative\nGives the gradient of cross-entropy through softmax as the difference between predicted and true distributions.\nThe Gram Matrix Is Positive Semi-Definite Also known as: Gram Matrix PSD, Positive Semi-Definite Kernel Matrix, Gram Matrix Property\nShows that any Gram matrix is positive semi-definite, so its eigenvalues are non-negative and it defines an inner product.\nThe KV Cache Is Exact Also known as: KV Cache Correctness, Exact Decoding Cache, Incremental Attention Equivalence\nArgues that caching keys and values during decoding leaves the attention output bit-for-bit unchanged rather than merely close.\nThe Polyak-Lojasiewicz Criterion Also known as: Polyak-Łojasiewicz Inequality, PL Condition, Gradient Domination\nGives the gradient-domination condition under which gradient descent converges linearly even when the objective is not convex.\nThe Variance Identity Also known as: Variance Identity, E[X²] − E[X]², Second-Moment Identity\nExpresses variance as a difference of moments, giving a form that holds for every admissible value of the variables.\nThe Woodbury Identity Also known as: Woodbury Matrix Identity, Matrix Inversion Lemma, Low-Rank Update Identity\nRewrites the inverse of a matrix plus a low-rank correction so that only a small system needs inverting.\n","permalink":"https://www.zhumengyao.com/scidraft/math-a-to-z/","summary":"\u003cp\u003eThis A–Z page lists every mathematical result covered in the \u003cstrong\u003eMath\u003c/strong\u003e note series.\nEach entry gives the result\u0026rsquo;s name, the family it belongs to, and a link to the\nnote that states or proves it.\u003c/p\u003e\n\u003cp\u003eTo browse by result family instead, visit \u003ca href=\"/math-categories/\"\u003eMath Categories\u003c/a\u003e.\u003c/p\u003e\n\n\n\u003cdiv class=\"entry-count\"\u003e\u003cstrong\u003e37 Results Covered\u003c/strong\u003e\u003c/div\u003e\n\u003cdiv class=\"entry-alphabet\"\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#A\"\u003eA\u003c/a\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#B\"\u003eB\u003c/a\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#C\"\u003eC\u003c/a\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eD\u003c/span\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#E\"\u003eE\u003c/a\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#F\"\u003eF\u003c/a\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#G\"\u003eG\u003c/a\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eH\u003c/span\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#I\"\u003eI\u003c/a\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eJ\u003c/span\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#K\"\u003eK\u003c/a\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#L\"\u003eL\u003c/a\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#M\"\u003eM\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eN\u003c/span\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eO\u003c/span\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#P\"\u003eP\u003c/a\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eQ\u003c/span\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#R\"\u003eR\u003c/a\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#S\"\u003eS\u003c/a\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#T\"\u003eT\u003c/a\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eU\u003c/span\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eV\u003c/span\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eW\u003c/span\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eX\u003c/span\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eY\u003c/span\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eZ\u003c/span\u003e\n  \u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv class=\"entry-map\"\u003e\n    \u003ch2 class=\"entry-letter\" id=\"A\"\u003eA\u003c/h2\u003e\n  \u003cdiv class=\"entry-item\"\u003e\n    \u003ca class=\"entry-title\" href=\"/scidraft/notes/demo-note-corollary-attention-norm/\"\u003eA Bound on the Attention Output Norm\u003c/a\u003e\n    \u003cp class=\"entry-alt\"\u003e\u003cem\u003eAlso known as:\u003c/em\u003e Attention Norm Bound, Output Norm Corollary, Bounded Attention Output\u003c/p\u003e","title":"Math A–Z"},{"content":"This page groups every mathematical result covered in the Math note series by result family. Within each family, results are listed alphabetically, with a link to the note that states or proves each one.\nFor the alphabetical view, visit Math A–Z.\n37 Results Covered in 20 Categories Algorithms Axioms Claims Conjectures Corollaries Counterexamples Criteria Definitions Examples Formulas Identities Lemmas Observations Postulates Principles Proofs Properties Propositions Rules Theorems Algorithms Gradient Descent with Momentum Also known as: Heavy Ball Method, Momentum Optimisation, EMA Gradient Descent\nKeeps a running average of past gradients so that descent accumulates speed along consistent directions and damps oscillation.\nMasked Softmax Attention Also known as: Causal Masking, Masked Attention, Autoregressive Softmax\nSpecifies how a causal mask is applied inside softmax attention so that each position attends only to itself and earlier positions.\nAxioms Inner-Product Axioms Also known as: Inner Product Space Axioms, Similarity Axioms, Positive-Definite Form\nFixes the three conditions a similarity measure must satisfy for attention scores to behave as inner products.\nKolmogorov Axioms for a Token Distribution Also known as: Kolmogorov Axioms, Probability Space Axioms, Normalised Non-Negative Measure\nStates the three assumptions — non-negativity, unit total mass, and additivity — that make a token distribution a probability distribution.\nClaims Temperature Cannot Sharpen Also known as: Temperature Monotonicity, No Sharpening Claim, Entropy Floor at Any Temperature\nAsserts that lowering the softmax temperature reweights probabilities without ever making the distribution more peaked than its logits allow.\nThe KV Cache Is Exact Also known as: KV Cache Correctness, Exact Decoding Cache, Incremental Attention Equivalence\nArgues that caching keys and values during decoding leaves the attention output bit-for-bit unchanged rather than merely close.\nConjectures Features Are Directions Also known as: Linear Representation Hypothesis, Feature Directions, Concept Vectors\nConjectures that a network\u0026#39;s learned concepts correspond to directions in activation space that can be added and compared linearly.\nPower-Law Scaling Also known as: Scaling Laws, Power-Law Scaling Hypothesis, Compute-Optimal Scaling\nHolds that loss falls as a power law in model size, data and compute, so performance can be extrapolated before training.\nCorollaries A Bound on the Attention Output Norm Also known as: Attention Norm Bound, Output Norm Corollary, Bounded Attention Output\nReads off a ceiling on the norm of an attention output as a direct consequence of the convex-combination proposition.\nPerplexity Is at Least One Also known as: Perplexity Lower Bound, Perplexity ≥ 1, Entropy Floor\nFollows from the definition of perplexity that its value can never drop below one, however good the model.\nCounterexamples A Convex Function with Many Minima Also known as: Non-Unique Optimum, Convex Counterexample, Flat Minimum Example\nShows by construction that convexity does not guarantee a unique minimiser, contradicting a common shorthand.\nMatrix Products Do Not Commute Also known as: Non-Commutativity of Matrices, AB ≠ BA, Order Dependence\nDemonstrates with two explicit matrices that reversing a product changes the result, so matrix order cannot be assumed interchangeable.\nCriteria Sylvester\u0026#39;s Criterion for Positive Definiteness Also known as: Sylvester\u0026#39;s Criterion, Leading Principal Minors Test, Positive Definiteness Test\nDecides positive definiteness by checking the signs of a matrix\u0026#39;s leading principal minors rather than inspecting its eigenvalues.\nThe Polyak-Lojasiewicz Criterion Also known as: Polyak-Łojasiewicz Inequality, PL Condition, Gradient Domination\nGives the gradient-domination condition under which gradient descent converges linearly even when the objective is not convex.\nDefinitions Cross-Entropy Also known as: Cross-Entropy Loss, Negative Log-Likelihood, Log Loss\nDefines the training objective of a language model as the expected negative log-probability assigned to the true tokens.\nScaled Dot-Product Attention Also known as: Softmax Attention, Query-Key-Value Attention\nFixes the notation for attention: three projections, one score matrix, and a softmax-weighted average over the values.\nExamples Attention with Two Tokens Also known as: Two-Token Attention Example, Minimal Attention Worked Example, Single-Head Example\nWorks the smallest interesting attention case — two positions, one head — end to end so every number can be checked by hand.\nSoftmax of Three Scores Also known as: Softmax Worked Example, Three-Score Softmax, Numerical Softmax Demo\nEvaluates softmax on three scores so the normalisation step is visible as arithmetic rather than notation.\nFormulas Rotary Position Embedding Also known as: RoPE, Rotary Positional Encoding\nEncodes position by rotating query and key vectors, so attention scores depend on relative distance rather than absolute index.\nThe Adam Update Also known as: Adam Optimiser, Adaptive Moment Estimation, Adam Update Rule\nWrites the Adam update in symbols, combining a momentum term with a running second-moment estimate and a bias correction.\nIdentities The Variance Identity Also known as: Variance Identity, E[X²] − E[X]², Second-Moment Identity\nExpresses variance as a difference of moments, giving a form that holds for every admissible value of the variables.\nThe Woodbury Identity Also known as: Woodbury Matrix Identity, Matrix Inversion Lemma, Low-Rank Update Identity\nRewrites the inverse of a matrix plus a low-rank correction so that only a small system needs inverting.\nLemmas Log-Sum-Exp Bounds Also known as: LSE Inequalities, Max-Plus Bounds\nBounds the log-sum-exp of a set of values between their maximum and that maximum plus a log-count term.\nSoftmax Is Invariant to a Shift Also known as: Softmax Shift Invariance, Translation Invariance of Softmax, Max-Subtraction Lemma\nEstablishes that adding a constant to every logit leaves softmax unchanged, which is what makes max-subtraction safe.\nObservations Attention Entropy Falls with Depth Also known as: Attention Entropy Decay, Entropy vs Depth, Attention Concentration\nReports the empirical finding that attention distributions become more concentrated in deeper layers.\nPostulates Smoothness of the Loss Also known as: Smoothness Assumption, Lipschitz Gradient Assumption, Bounded Curvature\nAssumes the loss has a bounded gradient Lipschitz constant, the condition most convergence proofs are stated under.\nPrinciples Equivariance as a Design Principle Also known as: Equivariance Principle, Symmetry as Inductive Bias, Group Equivariance\nTreats a symmetry of the problem as a design constraint, so that transforming the input transforms the output in the matching way.\nThe Bias-Variance Trade-Off Also known as: Bias-Variance Trade-Off, Bias-Variance Decomposition, Capacity Trade-Off\nStates that expected error decomposes into bias and variance, so added capacity trades one against the other.\nProofs Proof of the Softmax Shift Lemma Also known as: Softmax Shift Proof, Translation Invariance Proof, Max-Subtraction Correctness\nProves the shift-invariance lemma by factoring the added constant out of numerator and denominator.\nProof: Attention Outputs Lie in the Convex Hull Also known as: Convex Hull Proof, Attention as Convex Combination, Output Hull Membership Proof\nProves that an attention output is a convex combination of its value vectors, hence lies inside their convex hull.\nProperties Cross-Entropy Is Convex in the Logits Also known as: Convexity of Cross-Entropy, Log-Sum-Exp Convexity, Convex Loss Property\nEstablishes that cross-entropy is convex in the logits, which is why the training objective has no spurious local minima.\nThe Gram Matrix Is Positive Semi-Definite Also known as: Gram Matrix PSD, Positive Semi-Definite Kernel Matrix, Gram Matrix Property\nShows that any Gram matrix is positive semi-definite, so its eigenvalues are non-negative and it defines an inner product.\nPropositions Attention Outputs Are Convex Combinations Also known as: Attention Convexity Proposition, Convex Combination Proposition, Attention Weights Sum to One\nStates that attention weights are non-negative and sum to one, so each output is a convex combination of the values.\nThe Gradient of Cross-Entropy Through Softmax Also known as: Softmax Gradient Proposition, Cross-Entropy Gradient, Softmax-Cross-Entropy Derivative\nGives the gradient of cross-entropy through softmax as the difference between predicted and true distributions.\nRules Bayes\u0026#39; Rule Also known as: Bayes\u0026#39; Theorem, Bayes\u0026#39; Formula, Posterior Rule\nRelates a posterior to a likelihood and a prior, giving the rule for updating belief in light of evidence.\nThe Chain Rule Also known as: Chain Rule, Composite Differentiation Rule, Chain Rule of Calculus\nStates that the derivative of a composition is the product of the derivatives of its parts, the basis of backpropagation.\nTheorems Maximum Entropy Gives the Softmax Also known as: Maximum Entropy Principle, MaxEnt Derivation of Softmax, Entropy-Maximising Distribution\nDerives the softmax distribution as the unique maximiser of entropy subject to matching expected scores.\n","permalink":"https://www.zhumengyao.com/scidraft/math-categories/","summary":"\u003cp\u003eThis page groups every mathematical result covered in the \u003cstrong\u003eMath\u003c/strong\u003e note series by\nresult family. Within each family, results are listed alphabetically, with a link\nto the note that states or proves each one.\u003c/p\u003e\n\u003cp\u003eFor the alphabetical view, visit \u003ca href=\"/math-a-to-z/\"\u003eMath A–Z\u003c/a\u003e.\u003c/p\u003e\n\n\n\u003cdiv class=\"entry-count\"\u003e\u003cstrong\u003e37 Results Covered in 20 Categories\u003c/strong\u003e\u003c/div\u003e\n\n\u003cdiv class=\"entry-alphabet\"\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#algorithms\"\u003eAlgorithms\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#axioms\"\u003eAxioms\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#claims\"\u003eClaims\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#conjectures\"\u003eConjectures\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#corollaries\"\u003eCorollaries\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#counterexamples\"\u003eCounterexamples\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#criteria\"\u003eCriteria\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#definitions\"\u003eDefinitions\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#examples\"\u003eExamples\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#formulas\"\u003eFormulas\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#identities\"\u003eIdentities\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#lemmas\"\u003eLemmas\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#observations\"\u003eObservations\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#postulates\"\u003ePostulates\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#principles\"\u003ePrinciples\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#proofs\"\u003eProofs\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#properties\"\u003eProperties\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#propositions\"\u003ePropositions\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#rules\"\u003eRules\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#theorems\"\u003eTheorems\u003c/a\u003e\n  \u003c/div\u003e\n\u003c/div\u003e\n\n\u003cdiv class=\"entry-map\"\u003e\n  \u003ch2 class=\"entry-letter\" id=\"algorithms\"\u003eAlgorithms\u003c/h2\u003e\n  \u003cdiv class=\"entry-item\"\u003e\n    \u003ca class=\"entry-title\" href=\"/scidraft/notes/demo-note-algorithm-momentum/\"\u003eGradient Descent with Momentum\u003c/a\u003e\n    \u003cp class=\"entry-alt\"\u003e\u003cem\u003eAlso known as:\u003c/em\u003e Heavy Ball Method, Momentum Optimisation, EMA Gradient Descent\u003c/p\u003e","title":"Math Categories"},{"content":"This A–Z page lists every rendering method demonstrated in the Rendering note series. Each entry gives the method or diagram, the category it belongs to, and a link to the note that demonstrates it.\nTo browse by rendering category instead, visit Rendering Categories.\n23 Methods Covered A B C D E F G H I J K L M N O P Q R S T U V W X Y Z A A Paper with Multiple References Also known as: Multiple Bibliographies, Per-Paper References, Multi-Bib Note\nShows a single note carrying more than one reference list, each drawing on its own bibliography file.\nAttention Mechanism Pseudocode Also known as: Attention Pseudocode, Scaled Dot-Product Algorithm, Softmax Attention Procedure\nWrites scaled dot-product attention as pseudocode, making the matmul, scale, mask and weighted sum explicit.\nAttention Variants Table Also known as: Attention Variants, Long-Context Comparison, Attention Cost Table\nCompares attention variants for long contexts by cost and memory, in the striped table style.\nB Backpropagation Pseudocode Also known as: Backpropagation Algorithm, Reverse-Mode Differentiation, Gradient Computation Pseudocode\nStates backpropagation as a two-pass procedure: a forward pass caching activations and a reverse pass accumulating gradients.\nBenchmark Survey Also known as: Benchmark Survey, Third Bibliography, Benchmark References\nPairs a benchmark discussion with its own bibliography, demonstrating that bibliographies are named per note.\nC Causal Attention in Python Also known as: Causal Attention in Python, Masked Attention Code, Attention Implementation\nImplements causal attention in Python, applying a triangular mask before the softmax so no position reads ahead.\nCross-Entropy Gradient in Python Also known as: Cross-Entropy Gradient Code, Softmax Gradient Implementation, Log-Loss Backward Pass\nComputes the cross-entropy gradient through softmax in Python, showing the predicted-minus-true form in code.\nG Gradient Accumulation Also known as: Gradient Accumulation, Micro-Batching, Accumulated Update Procedure\nShows how to reach a large effective batch size by accumulating gradients over several micro-batches before stepping.\nL Layer Normalization Also known as: Layer Normalisation, LayerNorm Procedure, Per-Sample Normalisation\nGives layer normalisation as pseudocode, normalising each sample across its features and then rescaling with learned parameters.\nM Masked Softmax Attention Also known as: Masked Attention Table, Causal Mask Table, Attention Mask Reference\nTabulates how a causal mask constrains each attention position, alongside the algorithm that applies it.\nMermaid: Flowchart Also known as: Flowchart, Flow Diagram, Process Diagram\nTraces a transformer training step as a directed flowchart, showing the order of forward pass, loss and update.\nMermaid: Mindmap of Attention Also known as: Mindmap, Concept Map, Radial Diagram\nPresents the components of attention as a mindmap, grouping queries, keys, values and their relationships around a centre.\nMermaid: Overview Also known as: Mermaid Overview, Diagram Gallery, Mermaid Reference\nCollects the Mermaid diagram types the theme renders, each as a short working example.\nMermaid: Quadrant Chart Also known as: Quadrant Chart, Two-Axis Scatter, Priority Matrix\nPlots attention variants on two axes so their cost and cache requirements can be compared at a glance.\nMermaid: Sequence Diagram Also known as: Sequence Diagram, Message Sequence Chart, Interaction Diagram\nOrders the calls in an inference request as a sequence diagram, from client through the model to the token stream back.\nMermaid: State Diagram Also known as: State Diagram, State Machine, Transition Diagram\nShows a decoding loop as a state diagram, with the transitions between prompt, generation and stop states.\nMermaid: Training Gantt Chart Also known as: Gantt Chart, Training Schedule, Timeline Diagram\nLays a training schedule out as a Gantt chart, showing which stages run in parallel and where they wait on each other.\nN Notation Reference Table Also known as: Notation Reference, Symbol Table, Attention Notation\nTabulates the symbols used for multi-head attention — shapes and meanings side by side — in the academic table style.\nP Power Iteration Also known as: Power Iteration, Power Method, Dominant Eigenvector Algorithm\nComputes a matrix\u0026#39;s dominant eigenvector by repeated multiplication, the method behind spectral norm estimation.\nS Softmax in Python Also known as: Softmax in Python, Stable Softmax Code, Softmax Implementation\nImplements a numerically stable softmax in Python, subtracting the maximum before exponentiating.\nSurvey: Chain of Thought Also known as: Chain-of-Thought Survey, Reasoning Survey, CoT Literature Review\nSurveys chain-of-thought prompting, combining prose, a pseudo-algorithm, a sequence diagram and a bibliography.\nT Top-p Sampling Also known as: Nucleus Sampling, Top-p Sampling, Nucleus Decoding\nSelects the smallest set of tokens whose cumulative probability reaches p, then samples from it rather than from the full distribution.\nV Validation Loss by Model Size Also known as: Loss by Model Size, Scaling Results Table, Validation Loss Table\nSets validation loss against model size, in the bordered table style, as the tabular form of a scaling result.\n","permalink":"https://www.zhumengyao.com/scidraft/rendering-a-to-z/","summary":"\u003cp\u003eThis A–Z page lists every rendering method demonstrated in the \u003cstrong\u003eRendering\u003c/strong\u003e note\nseries. Each entry gives the method or diagram, the category it belongs to, and a\nlink to the note that demonstrates it.\u003c/p\u003e\n\u003cp\u003eTo browse by rendering category instead, visit \u003ca href=\"/rendering-categories/\"\u003eRendering Categories\u003c/a\u003e.\u003c/p\u003e\n\n\n\u003cdiv class=\"entry-count\"\u003e\u003cstrong\u003e23 Methods Covered\u003c/strong\u003e\u003c/div\u003e\n\u003cdiv class=\"entry-alphabet\"\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#A\"\u003eA\u003c/a\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#B\"\u003eB\u003c/a\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#C\"\u003eC\u003c/a\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eD\u003c/span\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eE\u003c/span\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eF\u003c/span\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#G\"\u003eG\u003c/a\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eH\u003c/span\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eI\u003c/span\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eJ\u003c/span\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eK\u003c/span\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#L\"\u003eL\u003c/a\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#M\"\u003eM\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#N\"\u003eN\u003c/a\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eO\u003c/span\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#P\"\u003eP\u003c/a\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eQ\u003c/span\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eR\u003c/span\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#S\"\u003eS\u003c/a\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#T\"\u003eT\u003c/a\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eU\u003c/span\u003e\n      \u003ca class=\"entry-alphabet-letter active\" href=\"#V\"\u003eV\u003c/a\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eW\u003c/span\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eX\u003c/span\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eY\u003c/span\u003e\n      \u003cspan class=\"entry-alphabet-letter inactive\"\u003eZ\u003c/span\u003e\n  \u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv class=\"entry-map\"\u003e\n    \u003ch2 class=\"entry-letter\" id=\"A\"\u003eA\u003c/h2\u003e\n  \u003cdiv class=\"entry-item\"\u003e\n    \u003ca class=\"entry-title\" href=\"/scidraft/notes/demo-note-paper-with-multiple-references/\"\u003eA Paper with Multiple References\u003c/a\u003e\n    \u003cp class=\"entry-alt\"\u003e\u003cem\u003eAlso known as:\u003c/em\u003e Multiple Bibliographies, Per-Paper References, Multi-Bib Note\u003c/p\u003e","title":"Rendering A–Z"},{"content":"This page groups every rendering method demonstrated in the Rendering note series by category. Within each category, entries are listed alphabetically, with a link to the note that demonstrates the method.\nFor the alphabetical view, visit Rendering A–Z.\n23 Methods Covered in 5 Categories Citations Code Mermaid Pseudo-algorithm Tables Citations A Paper with Multiple References Also known as: Multiple Bibliographies, Per-Paper References, Multi-Bib Note\nShows a single note carrying more than one reference list, each drawing on its own bibliography file.\nBenchmark Survey Also known as: Benchmark Survey, Third Bibliography, Benchmark References\nPairs a benchmark discussion with its own bibliography, demonstrating that bibliographies are named per note.\nSurvey: Chain of Thought Also known as: Chain-of-Thought Survey, Reasoning Survey, CoT Literature Review\nSurveys chain-of-thought prompting, combining prose, a pseudo-algorithm, a sequence diagram and a bibliography.\nCode Causal Attention in Python Also known as: Causal Attention in Python, Masked Attention Code, Attention Implementation\nImplements causal attention in Python, applying a triangular mask before the softmax so no position reads ahead.\nCross-Entropy Gradient in Python Also known as: Cross-Entropy Gradient Code, Softmax Gradient Implementation, Log-Loss Backward Pass\nComputes the cross-entropy gradient through softmax in Python, showing the predicted-minus-true form in code.\nSoftmax in Python Also known as: Softmax in Python, Stable Softmax Code, Softmax Implementation\nImplements a numerically stable softmax in Python, subtracting the maximum before exponentiating.\nMermaid Mermaid: Flowchart Also known as: Flowchart, Flow Diagram, Process Diagram\nTraces a transformer training step as a directed flowchart, showing the order of forward pass, loss and update.\nMermaid: Mindmap of Attention Also known as: Mindmap, Concept Map, Radial Diagram\nPresents the components of attention as a mindmap, grouping queries, keys, values and their relationships around a centre.\nMermaid: Overview Also known as: Mermaid Overview, Diagram Gallery, Mermaid Reference\nCollects the Mermaid diagram types the theme renders, each as a short working example.\nMermaid: Quadrant Chart Also known as: Quadrant Chart, Two-Axis Scatter, Priority Matrix\nPlots attention variants on two axes so their cost and cache requirements can be compared at a glance.\nMermaid: Sequence Diagram Also known as: Sequence Diagram, Message Sequence Chart, Interaction Diagram\nOrders the calls in an inference request as a sequence diagram, from client through the model to the token stream back.\nMermaid: State Diagram Also known as: State Diagram, State Machine, Transition Diagram\nShows a decoding loop as a state diagram, with the transitions between prompt, generation and stop states.\nMermaid: Training Gantt Chart Also known as: Gantt Chart, Training Schedule, Timeline Diagram\nLays a training schedule out as a Gantt chart, showing which stages run in parallel and where they wait on each other.\nPseudo-algorithm Attention Mechanism Pseudocode Also known as: Attention Pseudocode, Scaled Dot-Product Algorithm, Softmax Attention Procedure\nWrites scaled dot-product attention as pseudocode, making the matmul, scale, mask and weighted sum explicit.\nBackpropagation Pseudocode Also known as: Backpropagation Algorithm, Reverse-Mode Differentiation, Gradient Computation Pseudocode\nStates backpropagation as a two-pass procedure: a forward pass caching activations and a reverse pass accumulating gradients.\nGradient Accumulation Also known as: Gradient Accumulation, Micro-Batching, Accumulated Update Procedure\nShows how to reach a large effective batch size by accumulating gradients over several micro-batches before stepping.\nLayer Normalization Also known as: Layer Normalisation, LayerNorm Procedure, Per-Sample Normalisation\nGives layer normalisation as pseudocode, normalising each sample across its features and then rescaling with learned parameters.\nPower Iteration Also known as: Power Iteration, Power Method, Dominant Eigenvector Algorithm\nComputes a matrix\u0026#39;s dominant eigenvector by repeated multiplication, the method behind spectral norm estimation.\nTop-p Sampling Also known as: Nucleus Sampling, Top-p Sampling, Nucleus Decoding\nSelects the smallest set of tokens whose cumulative probability reaches p, then samples from it rather than from the full distribution.\nTables Attention Variants Table Also known as: Attention Variants, Long-Context Comparison, Attention Cost Table\nCompares attention variants for long contexts by cost and memory, in the striped table style.\nMasked Softmax Attention Also known as: Masked Attention Table, Causal Mask Table, Attention Mask Reference\nTabulates how a causal mask constrains each attention position, alongside the algorithm that applies it.\nNotation Reference Table Also known as: Notation Reference, Symbol Table, Attention Notation\nTabulates the symbols used for multi-head attention — shapes and meanings side by side — in the academic table style.\nValidation Loss by Model Size Also known as: Loss by Model Size, Scaling Results Table, Validation Loss Table\nSets validation loss against model size, in the bordered table style, as the tabular form of a scaling result.\n","permalink":"https://www.zhumengyao.com/scidraft/rendering-categories/","summary":"\u003cp\u003eThis page groups every rendering method demonstrated in the \u003cstrong\u003eRendering\u003c/strong\u003e note\nseries by category. Within each category, entries are listed alphabetically, with\na link to the note that demonstrates the method.\u003c/p\u003e\n\u003cp\u003eFor the alphabetical view, visit \u003ca href=\"/rendering-a-to-z/\"\u003eRendering A–Z\u003c/a\u003e.\u003c/p\u003e\n\n\n\u003cdiv class=\"entry-count\"\u003e\u003cstrong\u003e23 Methods Covered in 5 Categories\u003c/strong\u003e\u003c/div\u003e\n\n\u003cdiv class=\"entry-alphabet\"\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#citations\"\u003eCitations\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#code\"\u003eCode\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#mermaid\"\u003eMermaid\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#pseudo-algorithm\"\u003ePseudo-algorithm\u003c/a\u003e\n  \u003c/div\u003e\n  \u003cdiv class=\"entry-alphabet-line\"\u003e\n    \u003ca class=\"entry-alphabet-letter active\" href=\"#tables\"\u003eTables\u003c/a\u003e\n  \u003c/div\u003e\n\u003c/div\u003e\n\n\u003cdiv class=\"entry-map\"\u003e\n  \u003ch2 class=\"entry-letter\" id=\"citations\"\u003eCitations\u003c/h2\u003e\n  \u003cdiv class=\"entry-item\"\u003e\n    \u003ca class=\"entry-title\" href=\"/scidraft/notes/demo-note-paper-with-multiple-references/\"\u003eA Paper with Multiple References\u003c/a\u003e\n    \u003cp class=\"entry-alt\"\u003e\u003cem\u003eAlso known as:\u003c/em\u003e Multiple Bibliographies, Per-Paper References, Multi-Bib Note\u003c/p\u003e\n    \u003cp class=\"entry-desc\"\u003eShows a single note carrying more than one reference list, each drawing on its own bibliography file.\u003c/p\u003e","title":"Rendering Categories"}]