Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

A rule is a procedure applied so often that it stops looking like a theorem, and the chain rule is the extreme case: every backward pass through a stacked model is one long application of it.

Stated for compositions of two functions and applied left to right through a network, it converts the derivative of the loss with respect to the inputs into a product of local derivatives, each computed at the activations that were stored during the forward pass.

Rule (chain rule). If \(f\) is differentiable at \(g(x)\) and \(g\) is differentiable at \(x\), then the composition is differentiable and

For \(n\) composed functions \(y = f_n(f_{n-1}(\cdots f_1(x)\cdots))\) the rule is applied in turn, giving

Each factor is evaluated at the intermediate value the forward pass produced, which is why backpropagation stores activations rather than recomputing them.

The product form also explains a failure mode: when the factors are consistently below one in magnitude, the product shrinks geometrically with depth, which is the mechanism behind vanishing gradients in a recurrent or very deep model.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

The smallest useful project around a piece of neural-network mathematics holds one idea per file and keeps the generated numbers out of the code. Nothing in it is conventional yet: a starter folder is decided by what you want to compare next, not by what a template says.

The tree below is that starter size — a few files at the top level and one folder for data. Each entry carries a short note after #, and the highlighter treats those as comments, so the structure reads first and the explanations stay out of the way.

softmax-lab/
├── data/
│   └── logits.npy          # one batch of scores to experiment with
├── softmax.py              # the stable version: max-shift, then exp
├── temperature.py          # the same softmax, with a temperature
├── check_overflow.py       # naive vs shifted, on scores up to 1e3
└── README.md

Keeping the generated data/ folder separate from the scripts is the one habit worth adopting at this size. It is also the habit that runs out first: once a second model variant arrives, the flat layout forces the choice between longer file names and imports that depend on where the script was launched from — the reason the next size up introduces folders.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Backpropagation computes, for one layer, three gradients: one for each parameter of the layer and one to hand to the layer before it. The third is what makes deep networks trainable — without it, error could not travel past the last layer. The procedure below is the backward pass of a single dense layer.

The arithmetic follows from the chain rule. For \(Y = XW + b\), the gradient with respect to the weights contracts the input with the incoming gradient, \(\partial L/\partial W = X^{\top}\,\partial L/\partial Y\), while the bias gradient is the incoming gradient summed over the batch. The gradient passed backwards multiplies by the transposed weights, \(\partial L/\partial X = \partial L/\partial Y\, W^{\top}\), so each layer receives exactly the shape it produced during the forward pass.

\begin{algorithm}[H]
\caption{Backward Pass Through a Dense Layer}
\begin{algorithmic}[1]

\Require Layer input \(X \in \mathbb{R}^{n \times d_{in}}\)
\Require Weights \(W \in \mathbb{R}^{d_{in} \times d_{out}}\), bias \(b \in \mathbb{R}^{d_{out}}\)
\Require Upstream gradient \(\frac{\partial L}{\partial Y} \in \mathbb{R}^{n \times d_{out}}\)

\Ensure Parameter gradients \(\frac{\partial L}{\partial W}\), \(\frac{\partial L}{\partial b}\) and the gradient \(\frac{\partial L}{\partial X}\)

\State Recover the cached forward quantity: \(Y \leftarrow XW + b\)
\State Weight gradient: contract the input with the incoming gradient
\State \hspace{0.5cm} \(\frac{\partial L}{\partial W} \leftarrow X^{\top} \frac{\partial L}{\partial Y}\)
\State Bias gradient: sum the incoming gradient over the batch
\State \hspace{0.5cm} \(\frac{\partial L}{\partial b} \leftarrow \sum_{i=1}^{n} \frac{\partial L}{\partial Y}[i, :]\)
\State Gradient for the previous layer: apply the chain rule once more
\State \hspace{0.5cm} \(\frac{\partial L}{\partial X} \leftarrow \frac{\partial L}{\partial Y} W^{\top}\)
\Return \(\frac{\partial L}{\partial W}\), \(\frac{\partial L}{\partial b}\), \(\frac{\partial L}{\partial X}\)

\end{algorithmic}
\end{algorithm}

Reading the algorithm. Step 1 is a reminder that the forward quantity must have been cached: the backward pass reuses what the forward pass computed, which is why training needs memory proportional to depth. Steps 2–3 are two matrix multiplications — the same cost as the forward pass — and step 4 is the part that propagates error to earlier layers. Note the asymmetry with the forward pass: the forward pass multiplies by \(W\), the backward pass by \(W^{\top}\), and it is that transposition which keeps the shapes consistent in both directions.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Attention lets a model decide, for each position it is processing, which other positions to draw information from. This note states the scaled dot-product form as an algorithm rather than as an equation, so the sequence of operations — and the shape of each intermediate result — is explicit.

The scaling by \( \sqrt{d_k} \) matters: without it, the dot products grow with the key dimension, the softmax saturates, and the weights collapse onto a single position.

\begin{algorithm}[H]
\caption{Scaled Dot-Product Attention}
\begin{algorithmic}[1]

\Require Query matrix \(Q \in \mathbb{R}^{n \times d_k}\)
\Require Key matrix \(K \in \mathbb{R}^{m \times d_k}\)
\Require Value matrix \(V \in \mathbb{R}^{m \times d_v}\)

\Ensure Weighted output \(O \in \mathbb{R}^{n \times d_v}\)

\State Compute the similarity between every query and every key:
\State \hspace{0.5cm} \(S \leftarrow Q K^{\top}\) with \(S \in \mathbb{R}^{n \times m}\)

\State Scale the scores to keep the softmax in a stable range:
\State \hspace{0.5cm} \(S \leftarrow S / \sqrt{d_k}\)

\State Convert scores into weights that sum to one over the keys:
\State \hspace{0.5cm} \(A \leftarrow \mathrm{softmax}(S)\) rowwise

\State Read out a weighted combination of the values:
\State \hspace{0.5cm} \(O \leftarrow A V\)

\Return \(O\)

\end{algorithmic}
\end{algorithm}

Reading the algorithm. Steps 1–2 build the score matrix, where row \(i\) holds the compatibility of query \(i\) with every key; step 3 turns each row into a probability distribution, so a high score for one key means its value dominates that output row; step 4 mixes the values accordingly. Every row of the output therefore draws on a different combination of the same value matrix, which is what makes attention content-adaptive rather than fixed — at a cost of \(O(nm)\) for forming \(S\), the pressure behind the approximate and sparse variants.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Attention is built on a quantity that looks like a similarity but is defined by three axioms: an inner product is symmetric, linear in each argument, and positive definite. Those three properties are what allow a score to be read as “how much one vector points along another”.

Writing the axioms down is not pedantry here. The positive-definiteness clause is the one that rules out degenerate score functions, and the linearity clauses are what let the whole score matrix be computed as one matrix product.

Axiom (inner product). A function \(\langle \cdot, \cdot \rangle\) on two vectors is an inner product when

for all vectors and scalars, together with \(\langle x, x \rangle \gt 0\) for every non-zero \(x\). The score \(q^{\top} k\) satisfies all three, which is why attention is written as a product of matrices rather than as a table of pairwise scores.

Only the first two axioms are used when the scores are computed; positive definiteness is what guarantees that a vector has positive length, and it is invoked whenever a normalisation or a bound relies on \(\lVert x \rVert \gt 0\) for \(x \neq 0\).

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Position information has to enter attention somewhere, and one way is to rotate the query and key vectors by a position-dependent angle. The formula is appealing because the effect on the score depends only on the difference of the two positions.

That property is what the notation is chosen to display: a rotation preserves lengths and turns an inner product between rotated vectors into a function of the angle between them.

Formula (rotary embedding). Split a head vector \(x \in \mathbb{R}^{d_h}\) into \(d_h/2\) pairs \((x_{2i-1}, x_{2i})\). At position \(m\) and frequency \(\theta_i\), the rotated vector \(R_{\Theta, m} x\) has, in each pair,

Rotating both the query at position \(m\) and the key at position \(n\) gives an inner product that depends on \(m - n\) rather than on either position alone, because a rotation by \(m\theta\) composed with the inverse rotation by \(n\theta\) is a rotation by \((m-n)\theta\).

The price of the construction is that it encodes relative position only through rotation: a query and a key at the same position are unaffected, so the scheme says nothing about absolute location, which is either a feature or a limitation depending on the task.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

A scaling study compares models of different sizes trained on the same token budget. If the loss falls smoothly as capacity grows, the family is in a regime where parameters still buy accuracy, and the fall can be summarised as a power law in the parameter count rather than reported as a list of separate runs.

The table below is the theme’s default table style: every cell carries a border, the header row is tinted, and the numeric columns are right-aligned so that digits line up vertically. That alignment comes from the delimiter row of the Markdown table, not from the stylesheet, so an author can set it per column without touching any CSS.

Fitting the last four rows gives a loss that decays roughly as \(N^{-0.07}\), near the exponent reported for larger model families, so the shape of the curve is the expected one. The absolute numbers are not: the runs behind this demo used a fraction of the usual token budget, which is why the closing paragraph of a real note would carry the run configuration and the seed count instead.

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

An algorithm is a procedure with a specification, and this one says how the attention definition is executed on a machine: scale the scores, remove the positions that are not allowed, normalise and mix.

The steps are written in the order they must be performed. Masking before the softmax is the detail that matters — doing it afterwards would leave the removed positions contributing to the normalising constant and change the result.

Algorithm (masked softmax attention). Given \(Q, K, V \in \mathbb{R}^{T \times d_h}\) and a boolean mask \(M\), the steps are:

1. Compute the scaled scores \(S = QK^{\top} / \sqrt{d_h}\), of shape \(T \times T\).
2. Replace the scores that are not allowed to be attended to with \(-\infty\), that is \(S_{ij} \leftarrow -\infty\) wherever \(M_{ij}\) is true.
3. Subtract the row maximum: \(S_{ij} \leftarrow S_{ij} - \max_k S_{ik}\), which leaves the softmax unchanged and keeps the exponentials finite.
4. Exponentiate and normalise each row: \(A_{ij} = e^{S_{ij}} / \sum_k e^{S_{ik}}\).
5. Return \(A V\).

The cost is dominated by step one, which forms a \(T \times T\) matrix per head and therefore grows quadratically in the sequence length. Steps three and four are the numerical stabilisation from the log-sum-exp bounds.

Two of the five steps are present for reasons other than the mathematics: masking before normalising keeps the definition exact, and subtracting the row maximum keeps the arithmetic finite — the algorithm as written and the definition as stated compute the same function, and only one of the two is executable.