Gradients on SciDraft | A Hugo Theme

The Chain Rule (rule)

Mon, 21 Sep 2026 16:02:00 +0900

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

A rule is a procedure applied so often that it stops looking like a theorem, and the chain rule is the extreme case: every backward pass through a stacked model is one long application of it.

Softmax Cross-Entropy and Its Gradient in Python (advanced)

Mon, 21 Sep 2026 12:52:00 +0900

The number a language model reports while training is the mean negative log-likelihood of the token that actually came next: softmax cross-entropy over the vocabulary at every position. Its backward pass has an unusually clean closed form — the softmax Jacobian and the logarithm cancel, leaving the probabilities minus the one-hot target, divided by the number of predictions.

The Gradient of Cross-Entropy Through Softmax (proposition)

Mon, 21 Sep 2026 09:32:00 +0900

A proposition is a proved statement of moderate weight; this one is the reason the training loss needs no special treatment in the backward pass. Composing softmax with negative log-likelihood, the Jacobian of the softmax and the derivative of the logarithm cancel almost completely.