Formula on SciDraft | A Hugo Theme

Rotary Position Embedding (formula)

Mon, 21 Sep 2026 15:12:00 +0900

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Position information has to enter attention somewhere, and one way is to rotate the query and key vectors by a position-dependent angle. The formula is appealing because the effect on the score depends only on the difference of the two positions.

The Adam Update (formula)

Mon, 21 Sep 2026 11:12:00 +0900

A formula is a rule written in symbols so that it can be evaluated, and the one nearly every model is trained with is Adam’s update: two running averages of the gradient, a correction for the fact that both start at zero, and a step whose size is set by their ratio.