Source: Notebook: How LLMs & transformers work

The owner's interactive tutorial on tokens, embeddings, attention, the transformer block, next-token sampling and training.

Contents
  • What: interactive tutorial (owner’s Notebook site)
  • Author: Loic (coolloic/Notebook)
  • Date: accessed 2026-11-14
  • Original: How LLMs & transformers work

Summary

A conceptual walkthrough of the LLM forward pass for builders: text becomes tokens, tokens become position-aware vectors, stacked attention + feed-forward layers mix in context, and logits are turned into a sampled next token, repeated autoregressively. Ends with practical implications and a quiz.

Key claims

  • An LLM predicts the next token, appends it and repeats (autoregressive generation).
  • Tokens come from a fixed vocabulary of roughly 50k-200k entries; in English a token averages about 4 characters (~750 words per 1,000 tokens).
  • Self-attention lets each token gather context from others via queries, keys and values; multi-head attention learns several relation types in parallel.
  • A transformer layer is attention then feed-forward, with residuals and normalisation; dozens to 100+ layers are stacked.
  • Temperature, top-p/top-k and max tokens control sampling and length.
  • Builders must budget the context window (prompt + retrieved data + output) and pay per token.

Feeds

Transformer architecture · Tokenization · Context window · Sampling parameters

Suggest changes →