Attention Entropy Falls with Depth (observation)

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct. Entropy is a convenient scalar summary of an attention row, and the observation reported here is that its average falls as one moves up the layers: early heads spread their weight widely, later heads concentrate it. ...

September 21, 2026 · 2 min · yourname