Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

The matrix of pairwise inner products of a set of vectors has a property that constrains what similarity patterns are possible: it is symmetric and positive semi-definite, so its eigenvalues are non-negative and its diagonal entries are the squared lengths of the vectors.

The property is used in both directions. Forward, it says that a kernel matrix computed from a feature map is automatically a valid Gram matrix; backward, it says that an arbitrary table of similarities is not necessarily realisable by any embedding.

Property. For vectors \(x_1, \dots, x_n \in \mathbb{R}^{d}\), the Gram matrix \(G_{ij} = \langle x_i, x_j \rangle\) satisfies

for the matrix \(X\) with rows \(x_i\). Positive semi-definiteness follows from \(u^{\top} G u = \lVert X^{\top} u \rVert^{2} \geq 0\) for every \(u\), and it implies that every eigenvalue of \(G\) is non-negative and that \(\lvert G_{ij} \rvert \leq \sqrt{G_{ii} G_{jj}}\) by Cauchy-Schwarz.

The self-attention score matrix \(Q Q^{\top}\) is an instance: it records the similarity of each query with each other query, and its diagonal is the squared norm of each query.

Because the property is a constraint, it also serves as a test: a similarity table whose eigenvalues include a clearly negative one cannot have come from inner products of real vectors, so the feature map that supposedly produced it is mis-specified.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

The largest singular value of a weight matrix is one of the numbers used to judge how a layer transforms its input, and it is usually estimated rather than computed: repeated multiplication by the matrix makes any starting vector align with the leading eigenvector, after which the Rayleigh quotient gives the value.

This note uses the loop style of the pseudocode environment — a while with a stopping test and per-line comments. The stopping test compares the residual to the current eigenvalue estimate rather than to a fixed constant, so the same tolerance works at any scale of matrix.

\begin{algorithm}[H]
\footnotesize
\caption{Power iteration for the leading eigenvalue}
\begin{algorithmic}[1]
\Require symmetric matrix \(A\), start vector \(v\), tolerance \(\tau\)
\State \(v \gets v / \lVert v \rVert\) \Comment{any direction will do}
\State \(w \gets A v\); \(\lambda \gets \lVert w \rVert\) \Comment{first Rayleigh estimate}
\While{\(\lVert w - \lambda v \rVert \gt \tau \, \lambda\)}
    \State \(v \gets w / \lambda\) \Comment{renormalise, or the vector overflows}
    \State \(w \gets A v\); \(\lambda \gets \lVert w \rVert\)
\EndWhile
\Return \(\lambda, v\)
\end{algorithmic}
\end{algorithm}

The loop is written without a step limit, which is honest about the mathematics and optimistic in practice: the convergence rate is the ratio of the two largest eigenvalues, so an ill conditioned matrix can need many iterations, and an implementation normally adds a maximum step count to the stopping test for that reason.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Layer normalisation rescales each token’s activations using statistics computed across the features of that token alone, which is what separates it from batch normalisation: nothing in the computation depends on the other examples in the batch.

This note uses the function style of the pseudocode environment, with a named procedure, a parameter list and an explicit return. The mathematical content is three lines — a mean, a variance and an affine map — and the interesting part is the placement of the constant \(\epsilon\), which sits inside the square root so that a zero-variance input cannot divide by zero.

\begin{algorithm}[H]
\footnotesize
\caption{Layer normalisation of one token's activations}
\begin{algorithmic}[1]
\Function{LayerNorm}{\(x, \gamma, \beta, \epsilon\)}
    \State \(\mu \gets \frac{1}{d} \sum_{i=1}^{d} x_i\)
    \State \(\sigma^{2} \gets \frac{1}{d} \sum_{i=1}^{d} (x_i - \mu)^{2}\)
    \State \(\hat{x} \gets (x - \mu) / \sqrt{\sigma^{2} + \epsilon}\)
    \Return \(\gamma \odot \hat{x} + \beta\)
\EndFunction
\end{algorithmic}
\end{algorithm}

Two properties follow from the three lines. The mean and variance of \(\hat{x}\) are \(0\) and approximately \(1\) by construction, up to the effect of \(\epsilon\); and the learned scale and shift mean the layer as a whole can undo the normalisation if that is what the loss prefers, so the operation removes scale information only from the intermediate representation.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

A claim is a statement put forward as true and argued for, not yet established as a theorem. The one here concerns temperature scaling: raising the temperature flattens a distribution, and it cannot make the most probable token more probable than it already was.

The claim is worth stating because the opposite is sometimes assumed — that a large temperature might sharpen a peaky distribution. The argument below shows why the largest probability is monotone in the temperature, so no setting of it can increase that maximum.

Claim. Let \(z \in \mathbb{R}^{V}\) be a score vector and let \(p^{(T)} = \operatorname{softmax}(z / T)\) for \(T \gt 0\). Writing \(M(T) = \max_v p^{(T)}_v\), the claim is that

The argument: the temperature enters only through the ratio \(z / T\), which shrinks the spread of the scores as \(T\) grows, and the softmax is a monotone function of that spread. Equivalently, for \(T \geq 1\) the transformed scores \(z/T\) lie in the convex hull of the original scores and the origin, and the maximum probability of a distribution is a convex function of its scores.

The claim quantifies a lever rather than a guarantee: temperature cannot create confidence, it can only redistribute the confidence the scores already justify — which is why sampling at high temperature does not make a model more accurate, only more diverse.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Convexity of a loss in the parameters is rare in deep learning, but convexity in the scores is not, and the cross-entropy of a softmax has it. The property is what guarantees that the inner optimisation — fitting the output layer, probing a frozen representation — has no local minima to fall into.

The property is stated with respect to the logits, not the weights. As soon as the logits depend non-linearly on parameters, the composition need not be convex, which is exactly the boundary the statement respects.

Property. For a fixed target distribution \(p\), the function

is convex on \(\mathbb{R}^{V}\), and strictly convex on the affine subspace where the softmax is not constant. The derivative with respect to \(z\),

vanishes only where the predicted distribution equals the target, which is therefore the unique minimiser up to the shift redundancy of the softmax.

The property is why logistic regression is a convex problem and why fitting a linear probe on frozen features is reliable: in both cases the only non-convex part of the computation has been frozen, leaving an objective with a single basin.

DyCo Skills in one line: humans choose and oversee, agents draft and scale, the workflow is dynamic.

flowchart TD
    A[Start] --> B{Is the chart wide?}
    B -- yes --> C[Scroll horizontally]
    B -- no --> D[Fit the column]
    C --> E[End]
    D --> E

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Gradient accumulation is the standard way to train with a batch larger than memory allows: several small batches are processed in turn and their gradients added before any parameter moves. The gradient of a sum is the sum of the gradients, so the result equals one step on the concatenated batch as long as the per-batch losses are averaged.

This note writes the procedure in the contract style, stating what it requires and what it guarantees before any step is listed. Diagrams of the same loop usually omit the division by the number of micro-batches, which is the one detail that makes the accumulated gradient comparable to a single large-batch gradient.

\begin{algorithm}[H]
\footnotesize
\caption{Gradient accumulation over micro-batches}
\begin{algorithmic}[1]
\Require parameters \(\theta\), micro-batches \(B_1, \dots, B_k\), learning rate \(\eta\)
\Ensure parameters after one accumulated step
\State \(g \gets 0\)
\For{\(i = 1, \dots, k\)}
    \State evaluate the loss \(L_i\) on \(B_i\) with the current \(\theta\)
    \State \(g \gets g + \nabla_\theta L_i / k\)
\EndFor
\State \(\theta \gets \theta - \eta g\)
\Return \(\theta\)
\end{algorithmic}
\end{algorithm}

The contract is what makes the procedure checkable: the number of backward passes equals the number of micro-batches, the parameters move exactly once, and the accumulated gradient matches the mean gradient over the whole set of micro-batches to within the arithmetic of the summation order.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

A language model ends in a softmax: the raw scores of the last layer become a probability distribution over the vocabulary, and everything downstream — the training loss, sampling, beam search — reads that distribution rather than the scores themselves.

The snippet below is the smallest useful piece of that head: one function, one short array, and two printed lines. The line to notice is the shift before the exponential — subtracting the largest score leaves the distribution unchanged but keeps exp in range, so a score of 800 gives the same answer as a score of 0 instead of overflowing to infinity.

import numpy as np


def softmax(logits):
    """Turn a row of logits into a probability distribution."""
    shifted = logits - logits.max(axis=-1, keepdims=True)   # stays finite
    e = np.exp(shifted)
    return e / e.sum(axis=-1, keepdims=True)


logits = np.array([2.0, 1.0, 0.1])
p = softmax(logits)
print("probabilities", np.round(p, 3))
print("sum", p.sum())

Run as printed, it reports [0.659 0.242 0.099] and a sum of 1.0. Dividing the scores by a temperature before the softmax sharpens or flattens the same distribution, which is the knob that sampling turns.