Softmax

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. An example is where a formula stops being notation, and this one is small enough to check by hand: three scores, one softmax, three probabilities that must sum to one. The numbers below are the ones the companion snippet prints, so the arithmetic here and the output of a few lines of code can be compared directly. ...

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. A proof is the part of a note that cannot be skipped, and this one is short enough to show the whole pattern: write the definition, factor out the constant, use that it cancels between numerator and denominator. ...

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. A lemma is a small result proved on the way to something larger, and this one is proved in two lines yet carries most of the numerical stability of a softmax: adding the same constant to every score changes nothing. ...

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. A claim is a statement put forward as true and argued for, not yet established as a theorem. The one here concerns temperature scaling: raising the temperature flattens a distribution, and it cannot make the most probable token more probable than it already was. ...

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. A language model ends in a softmax: the raw scores of the last layer become a probability distribution over the vocabulary, and everything downstream — the training loss, sampling, beam search — reads that distribution rather than the scores themselves. ...