Pseudo-Algorithm on SciDraft | A Hugo Theme

Backpropagation Through a Dense Layer

Mon, 21 Sep 2026 15:42:00 +0900

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Backpropagation computes, for one layer, three gradients: one for each parameter of the layer and one to hand to the layer before it. The third is what makes deep networks trainable — without it, error could not travel past the last layer. The procedure below is the backward pass of a single dense layer.

Attention Mechanism: Pseudocode

Mon, 21 Sep 2026 15:32:00 +0900

Attention lets a model decide, for each position it is processing, which other positions to draw information from. This note states the scaled dot-product form as an algorithm rather than as an equation, so the sequence of operations — and the shape of each intermediate result — is explicit.

Nucleus (Top-p) Sampling

Mon, 21 Sep 2026 06:52:00 +0900

A language model does not emit a token; it emits a probability for every token in the vocabulary, and something has to choose one. Choosing the single most probable token makes output repetitive, while sampling from the full distribution occasionally picks from its long tail of near-zero probabilities. Nucleus sampling keeps the smallest set of tokens whose combined probability reaches a threshold and samples only from those.