Geometry on SciDraft | A Hugo Theme

Rotary Position Embedding (formula)

Mon, 21 Sep 2026 15:12:00 +0900

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Position information has to enter attention somewhere, and one way is to rotate the query and key vectors by a position-dependent angle. The formula is appealing because the effect on the score depends only on the difference of the two positions.

Features Are Directions (conjecture)

Mon, 21 Sep 2026 14:32:00 +0900

The linear representation hypothesis conjectures that the concepts a network has learned are encoded as directions in activation space, so that adding a concept vector to an activation moves the computation in a meaningful way.

A Bound on the Attention Output Norm (corollary)

Mon, 21 Sep 2026 13:02:00 +0900

Reading a bound off a proposition is the standard use of a corollary, and the one worth having here is a ceiling on how large an attention output can be. Since the output is a convex combination of value vectors, its length can never exceed the longest of them.

Equivariance as a Design Principle (principle)

Mon, 21 Sep 2026 11:52:00 +0900

Architectures are usually chosen for what they can compute, but the sharper criterion is what they cannot distinguish: matching the symmetries of the problem removes the need to learn them from data. This principle is why convolution appears in vision and why attention appears in sequences.