Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Long-context attention is usually chosen on cost rather than on quality. What decides whether a variant is usable at a given sequence length is how the score cost and the per-token cache grow with \(T\), and both are easiest to compare side by side in one table rather than across several paragraphs.

The table below uses the striped style ({.table-striped}): horizontal rules and a tinted every-other row, which keeps a row readable once it runs past five or six columns. Its cells hold plain text, typeset math, and a line break where two short answers belong in one cell, so a wide comparison still fits one screen.

The cache column is the one that decides serving cost. A full-attention layer keeps \(2 \cdot d_{\text{model}}\) numbers per token, so its memory grows with sequence length even when the score cost is no longer the bottleneck, while a state-space layer replaces that growth with a state whose size is fixed by design.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

A conjecture is a statement believed on the weight of evidence but not proved, and the best-known one in this field concerns how loss falls as models grow. Fits are tight over the measured range, and the conjecture is that the same form continues past it.

Nothing in the mathematics forces the form; it is adopted because the residuals of the fit are small and because the constants that come out are stable across independent studies. That is strong evidence and it is still not a proof.

Conjecture (power-law scaling). There exist constants \(L_\infty \geq 0\), \(N_c \gt 0\) and \(\alpha \gt 0\) such that, for models trained to convergence on a fixed token budget,

holds for every model size \(N\), and similarly in the number of training tokens \(D\) with its own exponent \(\beta\). The conjecture asserts the form outside the range where it has been fitted.

A consequence used in planning runs: doubling the parameter count reduces the size-dependent term by \(2^{-\alpha}\), so a small difference in \(\alpha\) compounds into a large difference in compute.

Because the statement is a conjecture, the honest way to use it is as a budget model: it says what a run is expected to buy, and any run whose measured loss sits well off the fitted curve is evidence about the form itself rather than only about that run.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Multi-head attention reads better as shape bookkeeping than as prose: every symbol below is a tensor, and the shapes alone decide which products are legal, which is why papers about attention variants usually open with a notation table before showing a single result.

The table below uses the academic three-rule style ({.table-academic}): a rule above the table, a thin rule under the header, a rule at the bottom, and no vertical lines — the shape journals and conference proceedings ask for. The style is chosen per table with a block attribute on the Markdown table, so a note can mix it with the default style without editing any stylesheet.

Shapes are written without the batch dimension. Adding it gives \(B \times T \times d_{\text{model}}\) for the inputs, and the per-head views become \(B \times H \times T \times d_{h}\) once each projection is split — the layout most implementations keep in memory, and the one that explains why the head count \(H\) costs nothing in total width as long as \(d_{h}\) shrinks with it.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Evaluating a system that uses tools needs its own body of sources: benchmarks for tool use, for long-context reasoning and for end-to-end knowledge work, together with the model reports those numbers are read against. This note keeps that literature in a bibliography of its own, separate from the prompting and fine-tuning lists used elsewhere on this site.

Tool-use competence is measured by MCP-Atlas \cite{bandi2026mcpatlas}, long-context reasoning by the AA-LCR benchmark \cite{aalcr}, and the translation result that most attention papers still compare against is the one in Bahdanau et al. \cite{bahdanau2014attention}.

The three entries above come from a file that shares no keys with the other bibliographies here, which is the point of naming a file per list: this note downloads only its own references, and a key that lives in another paper’s file would be reported as missing rather than silently resolved.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Convexity is often described as the property that guarantees a unique optimum, and the statement is false as written: convex functions can have a whole interval of minimisers. The counterexample is a flat-bottomed function, convex everywhere and constant on a segment.

The distinction that survives is between convex and strictly convex, and it matters in practice because a flat region is exactly where a gradient method stalls without having found an isolated point.

Counterexample. Define \(f : \mathbb{R} \to \mathbb{R}\) by

The function is convex, as a maximum of convex functions composed with the convex \(\lvert x \rvert\), and it is zero on the whole interval \([-1, 1]\). Every point of that interval is a global minimiser, so the minimiser set has more than one element; gradient descent started inside the interval sees \(f'(x) = 0\) and stops at whichever point it reached first.

Strict convexity rules this out: if \(f\) were strictly convex with a minimiser, that minimiser would be unique.

For a trained model the flat region is a shape to look for rather than a pathology: it means many parameter settings are equally good on the objective, and any preference among them has to come from regularisation rather than from the loss.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

A language model does not emit a token; it emits a probability for every token in the vocabulary, and something has to choose one. Choosing the single most probable token makes output repetitive, while sampling from the full distribution occasionally picks from its long tail of near-zero probabilities. Nucleus sampling keeps the smallest set of tokens whose combined probability reaches a threshold and samples only from those.

The threshold makes the candidate set adaptive rather than fixed: after a peaked distribution, the nucleus is one or two tokens; after a flat one, it can hold hundreds. Temperature acts first, dividing the logits before the softmax, so it reshapes the distribution that the nucleus is then cut from.

\begin{algorithm}[H]
\caption{Nucleus (Top-p) Sampling of One Token}
\begin{algorithmic}[1]

\Require Logits \(z \in \mathbb{R}^{V}\) for the next position
\Require Temperature \(T > 0\) and nucleus threshold \(p \in (0, 1]\)
\Require Random source \(u \sim \mathrm{Uniform}(0, 1)\)

\Ensure Sampled token index \(i^{\star}\)

\State Turn the logits into a distribution: \(q \leftarrow \mathrm{softmax}(z / T)\)
\State Sort the vocabulary by probability, largest first:
\State \hspace{0.5cm} \(q_{(1)} \geq q_{(2)} \geq \ldots \geq q_{(V)}\)
\State Find the smallest prefix whose probability mass reaches the threshold:
\State \hspace{0.5cm} \(k \leftarrow \min \{ m : \sum_{j=1}^{m} q_{(j)} \geq p \}\)
\State Renormalise the kept tokens so they sum to one:
\State \hspace{0.5cm} \(\tilde{q}_{(j)} \leftarrow q_{(j)} / \sum_{l=1}^{k} q_{(l)}\) for every \(j \leq k\)
\State Sample by walking the cumulative distribution with \(u\):
\State \hspace{0.5cm} \(i^{\star} \leftarrow \) the first index where the running sum of \(\tilde{q}\) exceeds \(u\)
\Return \(i^{\star}\)

\end{algorithmic}
\end{algorithm}

Reading the algorithm. Steps 1–2 produce a sorted distribution; step 3 is the whole idea — the size of the candidate set is a consequence of the model’s own confidence, not a constant. Step 4 matters for correctness rather than appearance: after truncation the probabilities no longer sum to one, so sampling without renormalisation would bias the choice towards the first kept token. The random draw in step 5 is why the same prompt can yield different completions, and why reproducibility requires fixing the seed as well as the temperature.