Geometry

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. Position information has to enter attention somewhere, and one way is to rotate the query and key vectors by a position-dependent angle. The formula is appealing because the effect on the score depends only on the difference of the two positions. ...

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. The linear representation hypothesis conjectures that the concepts a network has learned are encoded as directions in activation space, so that adding a concept vector to an activation moves the computation in a meaningful way. ...

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. Reading a bound off a proposition is the standard use of a corollary, and the one worth having here is a ceiling on how large an attention output can be. Since the output is a convex combination of value vectors, its length can never exceed the longest of them. ...

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. Architectures are usually chosen for what they can compute, but the sharper criterion is what they cannot distinguish: matching the symmetries of the problem removes the need to learn them from data. This principle is why convolution appears in vision and why attention appears in sequences. ...