Probability

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. Bayes’ rule is the rule for turning a model’s likelihood into a statement about hypotheses, and it is used twice in a language-model workflow: once to reason about uncertainty in evaluation, and once whenever a prior over prompts or parameters is combined with observed data. ...

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. An axiom is an assumption taken as given, and the ones that hold up a language model are Kolmogorov’s three: probabilities are non-negative, the whole space has probability one, and disjoint events add. Everything else about token distributions is derived from them rather than assumed. ...

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. The log-partition function appears wherever a normalising constant is needed, and it is trapped between two elementary bounds that are enough for most arguments: the largest term below, the largest term plus the log of the number of terms above. ...

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. The training objective of a language model has a name and a formula, and both matter when the loss curve is read: cross-entropy is the expected negative log-probability of what actually happened, measured in nats. ...