Pseudocode

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. The largest singular value of a weight matrix is one of the numbers used to judge how a layer transforms its input, and it is usually estimated rather than computed: repeated multiplication by the matrix makes any starting vector align with the leading eigenvector, after which the Rayleigh quotient gives the value. ...

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. Layer normalisation rescales each token’s activations using statistics computed across the features of that token alone, which is what separates it from batch normalisation: nothing in the computation depends on the other examples in the batch. ...

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. Gradient accumulation is the standard way to train with a batch larger than memory allows: several small batches are processed in turn and their gradients added before any parameter moves. The gradient of a sum is the sum of the gradients, so the result equals one step on the concatenated batch as long as the per-batch losses are averaged. ...