Transformer architecture

How a decoder-only transformer turns tokens into a next-token distribution: embeddings, causal self-attention, feed-forward blocks, positional encoding (RoPE) and the attention variants that make inference affordable.

Contents

The transformer is the neural-network architecture behind every current large language model. It was introduced in 2017 as an encoder-decoder for translation that drops recurrence and convolution entirely in favour of attention, a mechanism that lets every position in a sequence pull information from every other position in parallel (Vaswani et al.). Today’s LLMs use the decoder-only variant: a stack of identical blocks that reads a sequence of tokens and outputs a probability distribution over the next token. Generation repeats that one step: sample a token, append it, run again (notebook; Llama 3).

The forward pass

flowchart TD
    A["Text"] --> B["Tokenizer: text to token ids"]
    B --> C["Embedding lookup: id to vector (d_model)"]
    C --> D1
    subgraph BLK["Transformer block (repeated N times)"]
        D1["Norm"] --> D2["Causal multi-head self-attention (RoPE applied to Q, K)"]
        D2 --> D3["Add residual"]
        D3 --> D4["Norm"] --> D5["Feed-forward network (per position)"]
        D5 --> D6["Add residual"]
    end
    D6 --> E["Final norm + unembedding: vector to logits over vocabulary"]
    E --> F["Sampling: temperature / top-p / ..."]
    F --> G["Next token"]
    G -- "append and repeat (autoregressive)" --> B

Synthesis: block layout drawn in the pre-norm style common in modern decoder-only models; the 2017 paper placed normalisation after each residual add.

  1. Tokens to vectors. Each token id indexes a learned embedding table, giving a vector of width d_model (notebook).
  2. Blocks. Each block is self-attention followed by a position-wise feed-forward network, each wrapped in a residual connection and normalisation (Vaswani). Models stack dozens to over a hundred blocks; the learned weights in them are the model’s parameters (notebook). Modern decoders commonly use RMSNorm as the normalisation layer; it is one of the kernels Thinking Machines had to make batch-invariant in vLLM (Thinking Machines).
  3. Logits. The final vector at the last position is projected to one score (logit) per vocabulary entry; softmax turns them into probabilities, and the sampling parameters decide which token is picked.

Attention, precisely

For each position the block computes three projections of the input: a query (what this token is looking for), a key (what it offers to be matched on) and a value (what it contributes if matched). Scaled dot-product attention is

Attention(Q, K, V) = softmax(Q Kᵀ / √d_k) · V

The dot product of a query with every key gives a relevance score; dividing by √d_k stops scores from growing with dimension and saturating the softmax; the softmax weights then mix the values (Vaswani). Multi-head attention runs h of these in parallel on different learned projections (the original base model used h = 8, d_model = 512) so different heads can track different relations, such as syntax or coreference, then concatenates the results (Vaswani; notebook).

Causal masking makes the model a decoder: scores for future positions are set to minus infinity before the softmax, so position i only sees positions ≤ i. That is what allows training on every position of a sequence at once while still predicting left to right (Vaswani). Llama 3 additionally masks across document boundaries inside packed 8,192-token training sequences (Llama 3).

Cost. Attention connects any two positions in one step, but comparing every query with every key is quadratic in sequence length (Vaswani). Anthropic frames the consequence as an “attention budget”: n tokens create n² pairwise relationships that stretch thinner as context grows (Anthropic). See context window.

Positional encoding and RoPE

Attention is order-blind: shuffling the input permutes the output identically. Position must be injected.

SchemeHowWhere usedNotes
Sinusoidal (absolute)Fixed sin/cos vectors added to embeddingsOriginal TransformerSimple; no learned parameters (Vaswani)
RoPE (rotary)Rotate each Q and K vector by an angle proportional to its position, pairs of dimensions at different frequenciesMost modern open decodersQ·K then depends only on the relative offset; dependency decays with distance; flexible lengths (Su et al.)
RoPE + scaling (e.g. YaRN)Rescale RoPE frequencies, short fine-tuneLong-context variantsExtends context with about 10x fewer tokens and 2.5x fewer steps than prior methods (YaRN)

Why RoPE matters in practice: RoPE models do not generalise past the length they were trained on, because they meet rotation angles never seen in training (YaRN). Context extension by interpolation works, but positional precision degrades somewhat, which is one reason long-context quality declines gradually rather than falling off a cliff (Anthropic). Synthesis: this is a mechanism behind long-context degradation, and why an advertised window is not the same as a reliable one.

Making attention affordable at inference

Generation has two phases. Prefill processes the whole prompt in parallel. Decode produces one token at a time, and each step must read the keys and values of every earlier token, which are cached rather than recomputed (the KV cache). Decoding is therefore bound by memory bandwidth for loading K and V (GQA). Synthesis: this is why the variants below all target the K/V side.

VariantK/V headsEffect
Multi-head (MHA)One per query headBest quality, largest KV cache (GQA)
Multi-query (MQA)One sharedMuch faster decoding, can lose quality (GQA)
Grouped-query (GQA)One per group of query headsQuality close to MHA at near-MQA speed; used by Llama 3 8B and 70B (GQA; Llama 3)
Multi-head latent (MLA)K/V compressed into a latent vectorDeepSeek-V2: KV cache cut 93.3%, max throughput 5.76x vs DeepSeek 67B (DeepSeek-V2)

FlashAttention is orthogonal: it computes exact attention in tiles that fit in on-chip SRAM instead of writing the n×n score matrix to GPU memory, reporting 15% end-to-end training speedup on BERT-large (seq 512), 3x on GPT-2 (seq 1K) and 2.4x on Long-Range Arena (seq 1K-4K), and making longer contexts practical (FlashAttention). It changes speed and memory, not the model’s outputs.

Architecture families

FamilyAttentionTypical use
Encoder-decoderBidirectional encoder, causal decoder with cross-attentionOriginal Transformer, translation (Vaswani)
Decoder-onlyCausal self-attention onlyChat LLMs, e.g. Llama 3 (Llama 3)
Encoder-onlyBidirectional, no generationSynthesis: BERT-style models, used for classification and embeddings

Synthesis: decoder-only won for general LLMs because a single next-token objective covers every text task and trains on all positions at once.

Failure modes and limits that come from the architecture

  • Next-token prediction, not truth. The model outputs plausible continuations, so it can hallucinate fluently (notebook).
  • Fixed knowledge. Weights are frozen after training; there is a knowledge cutoff and no learning from chats without external memory or retrieval (notebook).
  • Quadratic attention and context rot. Longer inputs cost more and are used less reliably (Anthropic). See context window.
  • Length generalisation. RoPE models break beyond trained length unless extended (YaRN).
  • Numerical nondeterminism. Kernel results depend on batch size, so identical requests can diverge on a busy server (Thinking Machines). See sampling parameters.
  • Token-level view of text. The model sees subword ids, not characters; see tokenization.

Why an AI engineer needs this

Synthesis:

Sources

Suggest changes →