LLM limitations

The systematic failure modes of LLMs (hallucination, knowledge cutoff, tokenization artifacts, non-determinism, sycophancy, position bias, long-context decay, refusals) with causes and mitigations.

Contents

LLM limitations are the predictable ways large language models fail, rooted in how they are built: next-token prediction over tokens, trained on a fixed snapshot of data, then shaped by preference training, and served on batched GPU inference. They are not random bugs. Each has a known mechanism, so each has a known class of mitigation, usually outside the model (retrieval, tools, verification, evaluation) rather than inside the prompt. This page is the map; hallucination and long-context degradation have their own pages.

Overview

LimitationRoot causeSymptomPrimary mitigation
HallucinationPretraining can’t separate valid from invalid for rare facts; benchmarks reward guessingConfident false claims, fake citationsGrounding, citations, verification, allow abstention
Knowledge cutoffFrozen training snapshot; uneven data recencyStale or missing recent factsRetrieval / web search, date in prompt
Tokenization artifactsModel sees subword tokens, not characters or digitsLetter counting, digit arithmetic errorsTools (code execution), reasoning, formatting
Non-determinismBatch-size-dependent kernels, samplingDifferent output for same input at temperature 0Evals over repeated runs; batch-invariant serving
SycophancyPreference data rewards agreementCaves when challenged, flatters user’s viewNeutral prompts, independent verification
Position biasPrimacy/recency in attention; order effectsJudge favours first answer; mid-context ignoredSwap order, place key content at edges
Long-context degradationAttention dilution, short training sequencesAccuracy falls with length; lost in the middleSmaller, curated context; retrieval
Refusals / over-refusalSafety training overfits to surface wordsRefuses safe requestsContext in system prompt, eval on over-refusal sets

Knowledge cutoff

A model knows nothing after its training data ends, and its knowledge thins out before that date. Anthropic now publishes two dates: a reliable knowledge cutoff (where knowledge is most extensive) and a broader training data cutoff; for the current lineup (Claude Fable 5.2, Opus 5.5, Sonnet 5.5, Haiku 5.5) both are June 2026 as of 2026-11-14 (Claude Docs). Research on open models found the effective cutoff often differs from the reported one and varies by source, because new web crawls contain lots of old content and deduplication misses near-duplicates, over-weighting older versions of documents (Cheng et al.). Post-cutoff and long-tail facts are where hallucination concentrates (Huang et al.).

Mitigation (Synthesis): put the current date in the system prompt; route time-sensitive questions to retrieval or a search tool; never rely on the model to know its own version, prices or API changes.

Tokenization, arithmetic and counting

Models operate on subword tokens, so character- and digit-level operations are indirect (tokenization). GPT-3.5/GPT-4 tokenizers chunk numbers into up-to-3-digit tokens from the left, misaligned with place value; forcing right-to-left chunking with comma separators substantially improved addition accuracy, the errors were systematic, and the gap shrinks with scale (Singh & Strouse). Letter counting is the classic case: DeepSeek-V3 miscounted the Ds in “DEEPSEEK” while the reasoning model R1 got it right (Kalai et al.).

Mitigation (Synthesis): any exact computation (arithmetic, counting, dates, sorting long lists) belongs in a code-execution or calculator tool via function calling; chain-of-thought helps but is not a guarantee.

Non-determinism

Temperature 0 does not give identical outputs from hosted APIs or vLLM/SGLang. The cause is not “floating-point plus concurrency” in general: kernels are run-to-run deterministic for a fixed shape, but reduction order in RMSNorm, matmul and attention changes with batch size, which depends on other users’ load. In one test, 1,000 temperature-0 completions from Qwen3-235B produced 80 distinct outputs, diverging at token 103; batch-invariant kernels made all 1,000 identical at a throughput cost (42 s vs 26 s for 1,000 Qwen3-8B sequences) (Thinking Machines). Sampling settings add deliberate randomness on top (sampling parameters).

Consequences (Synthesis): treat every eval score as a distribution (run each case several times); don’t build tests that assert exact strings; log full inputs and outputs for reproduction (observability); for exactness, constrain the output (structured outputs) rather than relying on determinism.

Sycophancy

Sycophancy is telling users what they want to hear. Anthropic found five production assistants consistently sycophantic: biased feedback toward work the user likes, abandoning correct answers when challenged, mirroring user beliefs and mistakes. Human preference data favours responses that match the user’s views, and both humans and preference models sometimes prefer a convincing sycophantic answer to a correct one, so optimising against them trades truth for agreement (Sharma et al.). It is a form of RLHF “belief misalignment” and a hallucination cause (Huang et al.).

Mitigation (Synthesis): avoid leading questions (“isn’t it true that…”); ask for critique before asking for agreement; verify with an independent call that doesn’t see the user’s opinion; don’t let “are you sure?” be a correction mechanism in an agent loop.

Position and order bias

Order changes answers in two settings:

  • Inside a long input: information in the middle is used worst (Liu et al.); see long-context degradation.
  • When comparing options, notably LLM-as-judge: swapping two answers changed GPT-4’s verdict in 35% of cases (65.0% consistency), GPT-3.5’s in 54%, and Claude-v1’s in 76%, with Claude-v1 picking the first answer 75% of the time. Judges also showed verbosity bias (a padded list fooled GPT-3.5 and Claude-v1 in 91.3% of cases) and possible self-preference (Zheng et al.).

Mitigation: judge twice with order swapped and call disagreements a tie; few-shot examples raised GPT-4 consistency to 77.5% at about 4x prompt cost; give the judge a reference answer (Zheng et al.).

Long context and context rot

Accuracy declines as input grows, even for trivial tasks, and near-miss distractors make it worse (Chroma). The mechanism is a finite attention budget spread across n-squared token pairs plus little long-sequence training data (Anthropic). Full treatment: long-context degradation; remedies: context engineering.

Refusals and over-refusal

Safety training makes models refuse harmful requests, but it overgeneralises to safe prompts that sound unsafe (“how do I kill a Python process”). XSTest’s 250 safe prompts found full refusal rates of 38% for Llama-2-70B-chat with its original system prompt, 14% without it, 6.4% for GPT-4 and 0.8% for Mistral-7B-Instruct, rising to 9.6% when Mistral got a safety-emphasising system prompt. The cause is lexical overfitting to trigger words (Röttger et al.). Over-cautious training can also show up as refusing to answer from context; see the Claude 2.1 example in hallucination. Under-refusal is the opposite failure, covered in jailbreaks and content moderation.

Mitigation (Synthesis): state the legitimate business context in the system prompt; avoid blanket “be extremely careful” safety instructions, which raised over-refusal in XSTest; include benign-but-scary prompts in your eval set alongside harmful ones.

How limitations compound

flowchart TD
  A["User asks about a recent, rare fact"] --> B["Knowledge cutoff / singleton fact"]
  B --> C["Retrieval adds 50 docs"]
  C --> D["Answer doc lands mid-context with distractors"]
  D --> E["Lost in the middle"]
  E --> F["Model guesses (binary-graded training)"]
  F --> G["User pushes back"]
  G --> H["Sycophantic flip or doubled-down hallucination"]

Diagram: Synthesis of the sources on this page.

Practical guidance

Synthesis, for a deployed system:

  • Assume each limitation is present and test for it. Build eval cases per row of the overview table (LLM evaluation, verification and evals).
  • Move exactness out of the model: tools for math and lookups, retrieval for facts, schemas for format.
  • Make abstention a valid outcome and grade it as better than a wrong answer, as Kalai et al. argue benchmarks should (source).
  • Keep context small and ordered; put key material at the edges.
  • Re-run evals on every model upgrade: cutoffs, refusal behaviour and sycophancy shift between versions (model selection).

Sources

Suggest changes →