Math Categories

Also known as: Heavy Ball Method, Momentum Optimisation, EMA Gradient Descent

Keeps a running average of past gradients so that descent accumulates speed along consistent directions and damps oscillation.

Also known as: Causal Masking, Masked Attention, Autoregressive Softmax

Specifies how a causal mask is applied inside softmax attention so that each position attends only to itself and earlier positions.

Also known as: Inner Product Space Axioms, Similarity Axioms, Positive-Definite Form

Fixes the three conditions a similarity measure must satisfy for attention scores to behave as inner products.

Also known as: Kolmogorov Axioms, Probability Space Axioms, Normalised Non-Negative Measure

States the three assumptions — non-negativity, unit total mass, and additivity — that make a token distribution a probability distribution.

Also known as: Temperature Monotonicity, No Sharpening Claim, Entropy Floor at Any Temperature

Asserts that lowering the softmax temperature reweights probabilities without ever making the distribution more peaked than its logits allow.

Also known as: KV Cache Correctness, Exact Decoding Cache, Incremental Attention Equivalence

Argues that caching keys and values during decoding leaves the attention output bit-for-bit unchanged rather than merely close.

Also known as: Linear Representation Hypothesis, Feature Directions, Concept Vectors

Conjectures that a network's learned concepts correspond to directions in activation space that can be added and compared linearly.

Also known as: Scaling Laws, Power-Law Scaling Hypothesis, Compute-Optimal Scaling

Holds that loss falls as a power law in model size, data and compute, so performance can be extrapolated before training.

Also known as: Attention Norm Bound, Output Norm Corollary, Bounded Attention Output

Reads off a ceiling on the norm of an attention output as a direct consequence of the convex-combination proposition.

Also known as: Perplexity Lower Bound, Perplexity ≥ 1, Entropy Floor

Follows from the definition of perplexity that its value can never drop below one, however good the model.

Also known as: Non-Unique Optimum, Convex Counterexample, Flat Minimum Example

Shows by construction that convexity does not guarantee a unique minimiser, contradicting a common shorthand.

Also known as: Non-Commutativity of Matrices, AB ≠ BA, Order Dependence

Demonstrates with two explicit matrices that reversing a product changes the result, so matrix order cannot be assumed interchangeable.

Also known as: Sylvester's Criterion, Leading Principal Minors Test, Positive Definiteness Test

Decides positive definiteness by checking the signs of a matrix's leading principal minors rather than inspecting its eigenvalues.

Also known as: Polyak-Łojasiewicz Inequality, PL Condition, Gradient Domination

Gives the gradient-domination condition under which gradient descent converges linearly even when the objective is not convex.

Also known as: Cross-Entropy Loss, Negative Log-Likelihood, Log Loss

Defines the training objective of a language model as the expected negative log-probability assigned to the true tokens.

Also known as: Softmax Attention, Query-Key-Value Attention

Fixes the notation for attention: three projections, one score matrix, and a softmax-weighted average over the values.

Also known as: Two-Token Attention Example, Minimal Attention Worked Example, Single-Head Example

Works the smallest interesting attention case — two positions, one head — end to end so every number can be checked by hand.

Also known as: Softmax Worked Example, Three-Score Softmax, Numerical Softmax Demo

Evaluates softmax on three scores so the normalisation step is visible as arithmetic rather than notation.

Also known as: RoPE, Rotary Positional Encoding

Encodes position by rotating query and key vectors, so attention scores depend on relative distance rather than absolute index.

Also known as: Adam Optimiser, Adaptive Moment Estimation, Adam Update Rule

Writes the Adam update in symbols, combining a momentum term with a running second-moment estimate and a bias correction.

Also known as: Variance Identity, E[X²] − E[X]², Second-Moment Identity

Expresses variance as a difference of moments, giving a form that holds for every admissible value of the variables.

Also known as: Woodbury Matrix Identity, Matrix Inversion Lemma, Low-Rank Update Identity

Rewrites the inverse of a matrix plus a low-rank correction so that only a small system needs inverting.

Also known as: LSE Inequalities, Max-Plus Bounds

Bounds the log-sum-exp of a set of values between their maximum and that maximum plus a log-count term.

Also known as: Softmax Shift Invariance, Translation Invariance of Softmax, Max-Subtraction Lemma

Establishes that adding a constant to every logit leaves softmax unchanged, which is what makes max-subtraction safe.

Also known as: Attention Entropy Decay, Entropy vs Depth, Attention Concentration

Reports the empirical finding that attention distributions become more concentrated in deeper layers.

Also known as: Smoothness Assumption, Lipschitz Gradient Assumption, Bounded Curvature

Assumes the loss has a bounded gradient Lipschitz constant, the condition most convergence proofs are stated under.

Also known as: Equivariance Principle, Symmetry as Inductive Bias, Group Equivariance

Treats a symmetry of the problem as a design constraint, so that transforming the input transforms the output in the matching way.

Also known as: Bias-Variance Trade-Off, Bias-Variance Decomposition, Capacity Trade-Off

States that expected error decomposes into bias and variance, so added capacity trades one against the other.

Also known as: Softmax Shift Proof, Translation Invariance Proof, Max-Subtraction Correctness

Proves the shift-invariance lemma by factoring the added constant out of numerator and denominator.

Also known as: Convex Hull Proof, Attention as Convex Combination, Output Hull Membership Proof

Proves that an attention output is a convex combination of its value vectors, hence lies inside their convex hull.

Also known as: Convexity of Cross-Entropy, Log-Sum-Exp Convexity, Convex Loss Property

Establishes that cross-entropy is convex in the logits, which is why the training objective has no spurious local minima.

Also known as: Gram Matrix PSD, Positive Semi-Definite Kernel Matrix, Gram Matrix Property

Shows that any Gram matrix is positive semi-definite, so its eigenvalues are non-negative and it defines an inner product.

Also known as: Attention Convexity Proposition, Convex Combination Proposition, Attention Weights Sum to One

States that attention weights are non-negative and sum to one, so each output is a convex combination of the values.

Also known as: Softmax Gradient Proposition, Cross-Entropy Gradient, Softmax-Cross-Entropy Derivative

Gives the gradient of cross-entropy through softmax as the difference between predicted and true distributions.

Also known as: Bayes' Theorem, Bayes' Formula, Posterior Rule

Relates a posterior to a likelihood and a prior, giving the rule for updating belief in light of evidence.

Also known as: Chain Rule, Composite Differentiation Rule, Chain Rule of Calculus

States that the derivative of a composition is the product of the derivatives of its parts, the basis of backpropagation.

Also known as: Maximum Entropy Principle, MaxEnt Derivation of Softmax, Entropy-Maximising Distribution

Derives the softmax distribution as the unique maximiser of entropy subject to matching expected scores.