Source: Notebook: How LLMs & transformers work
The owner's interactive tutorial on tokens, embeddings, attention, the transformer block, next-token sampling and training.
Contents
- What: interactive tutorial (owner’s Notebook site)
- Author: Loic (coolloic/Notebook)
- Date: accessed 2026-11-14
- Original: How LLMs & transformers work
Summary
A conceptual walkthrough of the LLM forward pass for builders: text becomes tokens, tokens become position-aware vectors, stacked attention + feed-forward layers mix in context, and logits are turned into a sampled next token, repeated autoregressively. Ends with practical implications and a quiz.
Key claims
- An LLM predicts the next token, appends it and repeats (autoregressive generation).
- Tokens come from a fixed vocabulary of roughly 50k-200k entries; in English a token averages about 4 characters (~750 words per 1,000 tokens).
- Self-attention lets each token gather context from others via queries, keys and values; multi-head attention learns several relation types in parallel.
- A transformer layer is attention then feed-forward, with residuals and normalisation; dozens to 100+ layers are stacked.
- Temperature, top-p/top-k and max tokens control sampling and length.
- Builders must budget the context window (prompt + retrieved data + output) and pay per token.
Feeds
Transformer architecture · Tokenization · Context window · Sampling parameters