Python on SciDraft | A Hugo Theme

Causal Multi-Head Attention in Python (intermediate)

Tue, 22 Sep 2026 07:10:00 +0900

Demo note. This note demonstrates what the SciDraft Hugo theme can do for academic and scientific publishing — for research institutions, researchers and students. Its content is demonstration material only: readers should ignore whether the demo content is correct.

Attention is the one place in a transformer where tokens exchange information: every other block — the projections, the activations, the normalisations — treats positions independently. That makes attention mostly shape bookkeeping, and the shapes alone decide which products are legal.

Layout of a Scaling Study (advanced folder structure)

Tue, 22 Sep 2026 07:01:00 +0900

A project that produces a paper has to keep three kinds of thing apart that are easy to mix: the installed source, the outputs of individual runs, and the material the paper is written from. Once several sweeps and a performance benchmark live in the same repository, the folder structure is what decides whether a figure can be traced back to the run that produced it.

Layout of a Small Numeric Experiment (starter folder structure)

Mon, 21 Sep 2026 15:52:00 +0900

The smallest useful project around a piece of neural-network mathematics holds one idea per file and keeps the generated numbers out of the code. Nothing in it is conventional yet: a starter folder is decided by what you want to compare next, not by what a template says.

Softmax Cross-Entropy and Its Gradient in Python (advanced)

Mon, 21 Sep 2026 12:52:00 +0900

The number a language model reports while training is the mean negative log-likelihood of the token that actually came next: softmax cross-entropy over the vocabulary at every position. Its backward pass has an unusually clean closed form — the softmax Jacobian and the logarithm cancel, leaving the probabilities minus the one-hot target, divided by the number of predictions.

Layout of a Trainable Model (intermediate folder structure)

Mon, 21 Sep 2026 10:02:00 +0900

Once a model trains, the folder structure has a second job: it has to separate what changes per run from what changes per idea. Configurations and checkpoints belong to a run; attention code, data preparation and the training loop belong to the project, and the split is what lets a result be reproduced from a config file plus a commit rather than from memory.

Numerically Stable Softmax in Python (starter)

Mon, 21 Sep 2026 07:42:00 +0900

A language model ends in a softmax: the raw scores of the last layer become a probability distribution over the vocabulary, and everything downstream — the training loss, sampling, beam search — reads that distribution rather than the scores themselves.