Optimization on SciDraft | A Hugo Theme

A Training Schedule (gantt)

Mon, 21 Sep 2026 13:00:00 +0900

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

A training run has a shape in time as well as in the loss curve: data preparation that nothing else can start without, a warmup whose length is set by the step size, a long stable phase, and a cooldown whose purpose is to end at a point the optimiser has settled into.

Smoothness of the Loss (postulate)

Mon, 21 Sep 2026 12:32:00 +0900

Step sizes in deep learning are chosen as if the loss were smooth in its parameters, and that is an assumption, not a theorem. It is the smoothness postulate: the gradient does not change arbitrarily fast, so a small step in parameter space produces a predictable change in loss.

The Polyak-Lojasiewicz Criterion (criterion)

Mon, 21 Sep 2026 11:42:00 +0900

A criterion is a condition that decides whether an argument or a method applies, and this one is the usual substitute for convexity in the analysis of gradient descent: it bounds the loss by the size of its gradient, so a small gradient implies a small excess loss.

The Adam Update (formula)

Mon, 21 Sep 2026 11:12:00 +0900

A formula is a rule written in symbols so that it can be evaluated, and the one nearly every model is trained with is Adam’s update: two running averages of the gradient, a correction for the fact that both start at zero, and a step whose size is set by their ratio.

Gradient Descent with Momentum (algorithm)

Mon, 21 Sep 2026 10:32:00 +0900

The simplest optimiser with a memory is gradient descent with momentum: keep a running velocity, add the gradient to it, and step in the direction of the velocity rather than the gradient. The algorithm is short, and the behaviour it introduces is worth the two extra lines.

Cross-Entropy Is Convex in the Logits (property)

Mon, 21 Sep 2026 08:02:00 +0900

Convexity of a loss in the parameters is rare in deep learning, but convexity in the scores is not, and the cross-entropy of a softmax has it. The property is what guarantees that the inner optimisation — fitting the output layer, probing a frozen representation — has no local minima to fall into.

Gradient Accumulation with an Input and Output Contract (pseudocode)

Mon, 21 Sep 2026 07:47:00 +0900

Gradient accumulation is the standard way to train with a batch larger than memory allows: several small batches are processed in turn and their gradients added before any parameter moves. The gradient of a sum is the sum of the gradients, so the result equals one step on the concatenated batch as long as the per-batch losses are averaged.

A Convex Function with Many Minima (counterexample)

Mon, 21 Sep 2026 07:02:00 +0900

Convexity is often described as the property that guarantees a unique optimum, and the statement is false as written: convex functions can have a whole interval of minimisers. The counterexample is a flat-bottomed function, convex everywhere and constant on a segment.