Optimization

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. A training run has a shape in time as well as in the loss curve: data preparation that nothing else can start without, a warmup whose length is set by the step size, a long stable phase, and a cooldown whose purpose is to end at a point the optimiser has settled into. ...

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. Step sizes in deep learning are chosen as if the loss were smooth in its parameters, and that is an assumption, not a theorem. It is the smoothness postulate: the gradient does not change arbitrarily fast, so a small step in parameter space produces a predictable change in loss. ...

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. A criterion is a condition that decides whether an argument or a method applies, and this one is the usual substitute for convexity in the analysis of gradient descent: it bounds the loss by the size of its gradient, so a small gradient implies a small excess loss. ...

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. A formula is a rule written in symbols so that it can be evaluated, and the one nearly every model is trained with is Adam’s update: two running averages of the gradient, a correction for the fact that both start at zero, and a step whose size is set by their ratio. ...

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. The simplest optimiser with a memory is gradient descent with momentum: keep a running velocity, add the gradient to it, and step in the direction of the velocity rather than the gradient. The algorithm is short, and the behaviour it introduces is worth the two extra lines. ...

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. Convexity of a loss in the parameters is rare in deep learning, but convexity in the scores is not, and the cross-entropy of a softmax has it. The property is what guarantees that the inner optimisation — fitting the output layer, probing a frozen representation — has no local minima to fall into. ...

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. Gradient accumulation is the standard way to train with a batch larger than memory allows: several small batches are processed in turn and their gradients added before any parameter moves. The gradient of a sum is the sum of the gradients, so the result equals one step on the concatenated batch as long as the per-batch losses are averaged. ...

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. Convexity is often described as the property that guarantees a unique optimum, and the statement is false as written: convex functions can have a whole interval of minimisers. The counterexample is a flat-bottomed function, convex everywhere and constant on a segment. ...