Wiki

An LLM-maintained knowledge base on cutting-edge AI.

A knowledge base on cutting-edge AI, maintained by an LLM from sources I collect. Each concept page cites its sources. See the log for what changed and when.

Concepts

Forward deployed engineering

  • AI Governance and Compliance: The regulatory and standards layer for enterprise AI: EU AI Act obligations and dates, NIST AI RMF and its GenAI profile, ISO/IEC 42001, SOC 2 assurance, and bank model risk management, with how they fit together.
  • AI Use-Case Discovery: How to find, qualify, scope and prioritise LLM use cases in an organisation: where to look, fit tests for LLMs and agents, impact/effort scoring, success metrics and build-vs-buy.
  • Data Retention and Residency: What zero data retention (ZDR) and data residency actually guarantee on Anthropic, OpenAI, Google (Gemini API and Vertex AI), Azure and Bedrock: which stores survive ZDR, which features break it, how regional processing differs from regional storage, and where DPAs and BAAs fit.
  • Enterprise LLM Deployment: Running LLMs inside an enterprise’s security perimeter: data retention and ZDR, residency, identity and RBAC, audit logging, PII handling, network isolation, and how to roll out and drive adoption.
  • Forward Deployed Engineering: The forward deployed engineer (FDE) role: a product-grade software engineer embedded with one customer to ship production systems, born at Palantir and now central to how OpenAI, Anthropic and AI startups deploy LLMs.
  • POC to Production: Taking an LLM proof of concept to a dependable production system: why most pilots stall, the stage gates (evals, guardrails, integration, rollout), human-in-the-loop design, change management and the post-launch data flywheel.

Models

  • Batch Processing: Asynchronous vendor batch APIs that run large request sets at about 50% off within a 24-hour window: limits, lifecycle, result handling and caching interplay across Anthropic, OpenAI and Gemini.
  • Computer-Use Agents: Agents that operate GUIs and browsers through screenshots, mouse and keyboard (Anthropic computer/browser use, OpenAI CUA, Playwright-based agents): the action loop, grounding, benchmarks, reliability, prompt-injection risk and sandboxing.
  • Context window: The maximum number of tokens a model can attend to in one request (input plus output), what counts against it, vendor limits as of 2026-11, overflow behaviour, and why the usable window is smaller than the advertised one.
  • Extended Thinking: Claude’s API mechanism for visible-but-summarized reasoning: adaptive thinking with effort (current) or a fixed budget_tokens (legacy), thinking blocks with signatures, and interleaved thinking between tool calls.
  • Frontier Model Landscape: The proprietary frontier model families (Anthropic Claude, OpenAI GPT, Google Gemini, xAI Grok) as of November 2026: tiers, context windows, prices, and how vendors differentiate.
  • Function Calling: The API protocol by which a model emits structured requests to run developer-defined tools, and how Anthropic, OpenAI and Gemini differ on schemas, tool choice and parallel calls.
  • Hallucination: When an LLM produces fluent output that is false or unsupported: its types, why training makes it inevitable, how to detect it, and how to reduce it.
  • LLM limitations: The systematic failure modes of LLMs (hallucination, knowledge cutoff, tokenization artifacts, non-determinism, sycophancy, position bias, long-context decay, refusals) with causes and mitigations.
  • LLM Non-Determinism: Why the same prompt at temperature 0 can return different completions: batch-size-dependent kernels, floating-point precision, MoE routing, serving changes and seeds; what each vendor guarantees; and how to engineer around it.
  • Long-context degradation: Why LLM accuracy falls as inputs grow and depends on where information sits: lost-in-the-middle, needle-in-a-haystack limits, effective context length and context rot.
  • Mixture of Experts (MoE): Sparse transformer architecture where a router sends each token to a few of many expert FFNs: compute follows active parameters, memory follows total parameters, and it now underlies almost every large open-weight model.
  • Model Selection: How to choose an LLM for a workload: criteria, eval-driven selection, effort tuning before model switching, tier routing and cascades, and cost per completed task.
  • Multimodal Models: Models that take images, documents and audio as input (and sometimes produce images or speech): how inputs become tokens, per-vendor limits and costs, image generation, speech models, and failure modes.
  • Open-Weight Models: LLMs whose trained weights you can download and run yourself (Qwen, DeepSeek, Kimi, Mistral, Meta Muse/Llama, Gemma, gpt-oss): architectures, sizes, licences and trade-offs as of November 2026.
  • Prompt Caching: Reusing the stored KV state of a repeated prompt prefix to cut input cost and latency: mechanics, breakpoints, TTLs and pricing across Anthropic, OpenAI and Gemini.
  • Reasoning Models: LLMs trained with reinforcement learning to produce a long hidden chain of thought before answering, trading tokens and latency for accuracy on verifiable, multi-step problems.
  • Sampling parameters: How temperature, top-k, top-p, min-p, penalties, stop sequences and max tokens shape what an LLM generates, how each vendor exposes them as of 2026-11 (Anthropic removed temperature/top_p/top_k), and why temperature 0 is still not deterministic.
  • Streaming: Incremental delivery of model output over server-sent events: each vendor’s event grammar, how tool calls and thinking stream, mid-stream errors and recovery.
  • Structured Outputs: Forcing model output to match a JSON Schema via grammar-constrained decoding, how it differs from JSON mode, and each vendor’s schema subset, limits and failure modes.
  • Test-Time Compute: Spending more inference compute per query (longer reasoning, many samples plus selection, search against a verifier) to raise accuracy: a scaling axis separate from model size.
  • Tokenization: How LLMs split text into subword tokens (BPE, byte-level BPE, Unigram, WordPiece, SentencePiece), why token counts differ across vendors, models and languages, and how to count them correctly.
  • Transformer architecture: How a decoder-only transformer turns tokens into a next-token distribution: embeddings, causal self-attention, feed-forward blocks, positional encoding (RoPE) and the attention variants that make inference affordable.
  • Voice Agents: Agents that converse in real-time speech: cascaded STT-LLM-TTS vs speech-to-speech vs full-duplex architectures, the latency budget, turn detection and interruptions, transports, tool calls and evaluation.

Prompting

  • Chain-of-thought: Prompting a model to generate intermediate reasoning before its answer, plus its extensions (zero-shot CoT, self-consistency, tree of thoughts) and how built-in thinking changed it.
  • Constrained Decoding: How grammar- and JSON-schema-constrained generation works inside the decoder (token masks, FSM and CFG compilation, XGrammar, llguidance, Outlines), its real latency and quality costs, provider strict-mode coverage and failure modes.
  • Few-shot prompting: Steering an LLM with a handful of input-output demonstrations in the prompt (in-context learning): how it works, how to pick and format examples, and its biases.
  • Instructor and Outlines: The two Python approaches to typed LLM output: Instructor validates Pydantic models client-side and re-asks on failure; Outlines constrains generation token by token (or delegates to provider strict modes). Mechanics, modes, costs, failure modes and when to use which.
  • Prompt engineering: Designing, structuring, testing and versioning the instructions and inputs an LLM sees in a single call, per Anthropic, OpenAI and Google guidance.
  • System prompts: The privileged, operator-level instructions that frame a conversation: how vendors implement them (system, developer, instructions), their precedence over user turns, what belongs in them, and their limits.

Retrieval and data

  • Agentic RAG: RAG where an LLM controls retrieval: deciding whether, what and how often to retrieve, grading results and correcting course; covers routers, corrective RAG (CRAG), Self-RAG, adaptive RAG and just-in-time agentic search.
  • ANN Indexes: Approximate nearest-neighbour index structures (HNSW, IVF, PQ and other quantizers, DiskANN) that trade a little recall for orders-of-magnitude faster vector search, and how to tune them.
  • BM25: The Okapi BM25 lexical ranking function: IDF times a saturating, length-normalized term frequency, tuned by k1 and b; the default scorer in Lucene-family engines and the baseline every neural retriever is measured against.
  • Chunking: Splitting documents into retrievable units for RAG: fixed, recursive, semantic, LLM and late chunking, the size/overlap trade-off, contextual chunk headers, and how to measure chunk quality.
  • Contextual Retrieval: Index-time enrichment that prepends document-level context (an LLM-written blurb or a templated chunk header) to every chunk before embedding and BM25 indexing; Anthropic measured 35-67% fewer retrieval failures.
  • Document Processing: Document AI for LLM systems: turning PDFs, scans and office files into structured text with layout detection, OCR and table-structure models (Docling, Unstructured, LlamaParse) or with vision-language models, plus how to choose, evaluate and fail safely.
  • Embeddings: Fixed-length vectors from an encoder model that place semantically similar inputs close together; the representation behind semantic search and RAG.
  • GraphRAG: RAG over an LLM-built knowledge graph with hierarchical community summaries, built for multi-hop and whole-corpus questions that flat top-k retrieval cannot answer; plus cheaper variants like LazyGraphRAG.
  • Hybrid search: Combining lexical BM25 and dense vector retrieval and fusing the two rankings, usually with reciprocal rank fusion (RRF) or a weighted score blend.
  • Late-Interaction Retrieval: Multi-vector retrieval (ColBERT, ColBERTv2/PLAID, ColPali/ColQwen for page images) that keeps one vector per token or image patch and scores with MaxSim: how it works, what it costs to store, how to scale it, and when it beats single-vector embeddings.
  • Query Transformation: Rewriting the user’s question before retrieval (rewriting, expansion, HyDE, multi-query fusion, step-back, decomposition, self-query metadata filters): how each works, what it costs, and when it helps or hurts.
  • Reranking: A second retrieval stage that re-scores a wide candidate set with a more expensive model (cross-encoder, late-interaction or LLM) to put the truly relevant chunks on top.
  • Retrieval-augmented generation (RAG): Grounding an LLM’s answer in documents fetched at query time: the ingest, retrieve, augment, generate pipeline, its add-ons (query rewriting, HyDE, contextual retrieval), its CAG alternative and how to evaluate it.
  • Text-to-SQL: Translating natural-language questions into executable SQL with LLMs: schema linking, value retrieval, candidate generation and self-correction, semantic layers as the governed alternative, the Spider/BIRD/Spider 2.0 benchmarks and why their numbers overstate production accuracy.
  • Vector Databases: Systems that store embeddings with metadata and answer filtered nearest-neighbour queries at scale; how pgvector, Pinecone, Weaviate, Qdrant, Milvus and Chroma differ.

Training and fine-tuning

  • Fine-tuning: Continuing to train a pre-trained LLM’s weights on task data (SFT, preference or RL) to change how it behaves; when to use it versus prompting or RAG, how to prepare data, and how it fails.
  • Knowledge Distillation: Training a small student model to imitate a larger teacher, from Hinton’s soft targets to LLM sequence-level, black-box, reasoning and on-policy distillation.
  • LLM Training Pipeline: How a frontier LLM is produced, stage by stage (data curation, pretraining, mid-training and long-context extension, SFT, preference tuning, RLVR, constitutional and spec-based safety training), and how each stage explains behaviour you see in production.
  • PEFT and LoRA: Parameter-efficient fine-tuning: freeze the base model and train a small adapter. How LoRA, QLoRA, DoRA and rsLoRA work, their key hyperparameters, and when they match full fine-tuning.
  • Preference Optimization: Post-training that pushes a model toward preferred or rewarded outputs: RLHF with PPO, DPO and its variants, GRPO and RL with verifiable rewards, and RLAIF / Constitutional AI.
  • Reinforcement Fine-Tuning: Post-training an LLM with reinforcement learning against a programmatic or model-based grader (RLVR, GRPO and its variants, managed RFT APIs), which is how reasoning models are trained and how narrow expert tasks are sharpened.
  • Scaling Laws: Empirical power laws linking LLM loss to parameters, training tokens and compute: Kaplan vs Chinchilla, the ~20 tokens/parameter rule, why production models are deliberately over-trained for cheaper inference, and the limits of extrapolation.
  • Synthetic Data Generation: Using LLMs to write training and evaluation data (self-instruct, evolution, persona and template prompting, teacher distillation, rejection sampling), plus the filtering, decontamination and licensing work that decides whether it helps or hurts.

Inference and serving

  • Hugging Face Ecosystem: The Hugging Face Hub and its libraries (transformers, datasets, PEFT, safetensors, huggingface_hub) plus its hosted inference (Inference Providers, Inference Endpoints): what each does, key APIs, model cards and licences, pitfalls.
  • KV Cache: The per-token attention keys and values an LLM stores during generation: how to size it, how engines page and share it (PagedAttention, prefix caching, RadixAttention), and how GQA, MLA and FP8 shrink it.
  • LLM Inference Serving: How LLMs are served at scale: prefill vs decode, latency/throughput metrics, continuous batching, PagedAttention, and the vLLM / SGLang / TensorRT-LLM / TGI engines.
  • Local LLMs: Running open-weight models on your own laptop, workstation or server with Ollama, llama.cpp, LM Studio or MLX: how the runtimes work, memory math, key settings, failure modes and when local beats an API.
  • Quantization: Storing and computing LLM weights, activations and KV cache in fewer bits (FP8, INT8, INT4): how the methods work (GPTQ, AWQ, SmoothQuant, LLM.int8, GGUF), what they cost in accuracy, and which to use where.
  • Semantic Caching: Serving a stored LLM response when a new prompt’s embedding is close enough to a cached one: mechanics, threshold tuning, false hits, invalidation, per-user scoping and side channels, how it differs from prompt caching, and realistic hit rates.
  • Speculative Decoding: Cutting decode latency by letting a cheap drafter propose several tokens that the target model verifies in one pass, with a rejection rule that keeps outputs identical in distribution; variants include draft models, n-gram, Medusa, EAGLE and MTP.
  • vLLM: The open-source LLM serving engine built on PagedAttention: how its V1 engine schedules, caches and parallelizes, what its OpenAI-compatible server does, which flags matter, and how it compares with SGLang and TensorRT-LLM.

Evaluation and observability

  • Agent Evaluation: Evaluating multi-step agents rather than single completions: outcome vs trajectory grading, tool-call checks, simulated environments and users, pass@k vs pass^k, and cost and turn budgets.
  • Arize Phoenix: Source-available, OpenTelemetry-native tracing and evaluation tool: OpenInference span conventions, phoenix-otel setup, structured-output LLM evaluators, datasets and experiments, free self-hosting, and how it compares with Braintrust, Langfuse and LangSmith.
  • Benchmarks: Public LLM benchmarks (MMLU, GPQA, HLE, SWE-bench, τ-bench, ARC-AGI, Arena): what each measures, how it is graded, saturation, contamination, and how to read leaderboards.
  • Braintrust: Hosted eval-and-observability platform built around immutable experiments, versioned datasets, reusable scorers and sampled online scoring of production logs; how it works, its architecture, pricing and when to pick it.
  • Cost and Latency Optimization: The levers that cut LLM application cost and latency (prompt caching, semantic caching, batching, routing and cascades, smaller models, token budgets, streaming, parallelism) and their trade-offs.
  • Error Analysis: The practitioner loop of reading LLM and agent traces, open-coding failures in free text, axial-coding them into a counted failure taxonomy, and turning the top categories into targeted evals.
  • Langfuse: Open-source, OpenTelemetry-based LLM engineering platform: tracing, prompt management, datasets, experiments, LLM-as-judge and CI gating, self-hostable on Postgres + ClickHouse.
  • LLM Evaluation: Designing evals for LLM apps and agents: golden datasets, code, model and human graders, offline vs online, regression suites, statistics and tooling.
  • LLM Observability: Tracing, metrics and feedback for LLM apps and agents: the span data model, OpenTelemetry GenAI conventions, what to log, and how LangSmith, Langfuse and Phoenix differ.
  • LLM-as-Judge: Using a language model to grade outputs: pointwise vs pairwise vs reference-guided modes, known biases, calibration against humans, and when not to use it.
  • OpenTelemetry GenAI Semantic Conventions: The vendor-neutral OTel vocabulary for LLM and agent telemetry: gen_ai.* spans, attributes, events and metrics, opt-in content capture, spec status (still Development, own repo since 2026), the OpenLLMetry and OpenInference dialects, and which backends ingest it.
  • Prompt Management: Treating prompts as versioned, deployable artifacts: registries with immutable versions and movable labels, caching and fallbacks, trace linkage, and the eval-gated promotion loop.
  • promptfoo: Open-source (MIT) CLI and library for declarative LLM evals and red teaming: a prompts × providers × tests matrix scored by assertions, with exit codes that gate CI.
  • RAGAS and DeepEval: The two most-used open-source LLM metric libraries: how their faithfulness, relevancy and context precision/recall metrics and G-Eval are computed, and why same-named metrics can disagree.
  • Retrieval Metrics: How to measure a retriever: hit rate, recall@k, precision@k, MRR, MAP, nDCG and RAGAS context precision/recall, what each one rewards, where implementations disagree, and how to build the labelled set they all need.

Agents

  • Agent design patterns: Reusable architectures for arranging one or many agents, and how to choose between them.
  • Agent guardrails: Sandboxing, permissions, hooks and input handling that bound what an agent can do: the sandbox vs approval-policy split, the lethal trifecta and Rule of Two, SDK permission pipelines and their gaps, scoped credentials and egress control.
  • Agent Handoffs: Transferring control of a conversation from one agent to another via a transfer tool call, as opposed to calling an agent as a tool; mechanics in OpenAI Agents SDK, LangGraph, ADK and Agent Framework, context passing and failure modes.
  • Agent Instruction Files: Repository and user Markdown files (AGENTS.md, CLAUDE.md, GEMINI.md, Cursor rules) that coding agents load into every session as standing guidance: discovery order, size caps, scoping, and evidence that less is more.
  • Agent loop: The core of every AI agent: a model choosing actions in a loop until the task is done, plus the stop conditions, state rules and limits a production loop needs.
  • Agent Memory: How agents retain information beyond one context window: short-term thread state, long-term episodic/semantic/procedural stores, the write and retrieval pipelines (Mem0, Letta/MemGPT, LangGraph store, Claude memory tool), and how memory fails.
  • Agent Skills: Folders of instructions, scripts and resources (SKILL.md) that an agent discovers by description and loads on demand through progressive disclosure; an open standard read by Claude, Codex, Gemini CLI and Cursor.
  • Code Execution Sandboxes: Isolated environments (provider code-interpreter tools, microVM or gVisor sandbox platforms, OS-level sandboxes) that let agents run model-generated code while bounding what it can touch, reach and consume.
  • Coding Agents: LLM agents that read, edit, run and verify code in a real repository through a harness of tools, instructions files and sandboxes (Claude Code, Codex, Cursor, Copilot, Devin).
  • Context Compaction: Replacing older conversation or agent history with a model-written summary as the window fills: client vs server-side implementations (Anthropic, OpenAI), tool-result clearing, context resets, failure modes and how to choose.
  • Context engineering: Deciding what goes into an agent’s finite context window, and when.
  • Deep Research Agents: Agents that spend minutes to tens of minutes planning, searching the web in parallel, reading, and synthesising a cited report; architecture, vendor implementations (OpenAI, Google, Anthropic, Perplexity), evaluation, cost and failure modes.
  • Durable Agent Execution: Running long-lived agents so they survive crashes, deploys and multi-day waits: journaling model and tool calls, replaying to resume, idempotent side effects, and the Temporal / LangGraph / Inngest / Vercel Workflow / Restate options.
  • Harness Engineering: Designing everything around an LLM that turns it into a working agent (instructions, tools, environment, context management, orchestration, feedback) and re-tuning that harness as models change.
  • Human-in-the-Loop: Pausing an agent for human approval, editing or input at risky points: durable interrupts, tool approval gates (approve / edit / reject), escalation triggers, reviewer UX and security, and how the major frameworks implement it.
  • Letta: Letta (formerly MemGPT), a platform for stateful agents that manage their own memory: in-context memory blocks or git-backed MemFS files, archival and conversation search, sleep-time/dreaming agents that consolidate memory in the background, and a persistent agent server.
  • Mem0: Mem0, a drop-in long-term memory layer for agents: an LLM extracts facts from conversations into vector, SQL and entity stores, and retrieval fuses semantic, keyword and entity signals. Covers the 2025 CRUD pipeline, the 2026 ADD-only redesign, graph memory, the benchmark disputes, and Platform vs OSS.
  • Model Context Protocol (MCP): Open standard for exposing tools, resources and prompts to any compatible agent.
  • ReAct pattern: ReAct (Yao et al., 2022): an LLM agent interleaves free-text reasoning with tool actions and their observations; the ancestor of today’s tool-calling agent loops.
  • Subagents: Delegating focused sub-tasks to agents with fresh context windows.
  • Tools: Functions an agent can call to act on the world; their design is interface design for a model: sharp descriptions, lean results, readable errors, a small loaded catalogue and untrusted outputs.
  • Verification and evals: Feedback from outside the model, for the agent at runtime and for the builder over time: tests, hooks, critics and citation checks; capability vs regression evals, pass@k vs pass^k, judge biases and online evals.
  • Web Search APIs for Agents: How agents get live web access: provider-native search tools, LLM-oriented search APIs (Tavily, Exa, Brave, Perplexity, SerpApi), page extraction (Firecrawl, Jina Reader), citation grounding, and the cost, freshness and security trade-offs.

Frameworks and SDKs

  • Agent Hooks and Middleware: Deterministic code the harness runs at fixed points in the agent loop (before/after model calls, around tool calls, at stop) to block, rewrite, inject or observe: Claude Code / Agent SDK hooks, LangChain v1 middleware, OpenAI Agents SDK guardrails and lifecycle hooks, Mastra processors, and how they differ.
  • AutoGen and Microsoft Agent Framework: Microsoft’s conversation-first multi-agent framework AutoGen (now archived) and its successor, Microsoft Agent Framework, which merges AutoGen with Semantic Kernel around agents and typed graph workflows.
  • Claude Agent SDK: Anthropic’s Python/TypeScript library that embeds Claude Code’s agent loop, built-in tools, permissions, sessions, hooks and subagents in your own process.
  • CrewAI: Standalone Python multi-agent framework: role-based Crews (agents, tasks, sequential/hierarchical process) for autonomy, and event-driven Flows for deterministic state and control.
  • DSPy: Stanford’s framework for programming rather than prompting LMs: typed signatures, composable modules, adapters that compile them to prompts, and optimizers (BootstrapFewShot, MIPROv2, GEPA, BootstrapFinetune) that tune prompts or weights against a metric.
  • Google ADK: Google’s open-source, multi-language Agent Development Kit: LLM agents, workflow agents and (since 2.0) a graph-based workflow runtime, with sessions/state, evaluation, A2A and Google Cloud deployment.
  • LangChain: Open-source Python/JS framework that wraps LLM providers, tools and retrievers behind one Runnable interface and, since v1.0 (Oct 2025), centres on create_agent plus middleware running on LangGraph.
  • LangGraph: Low-level, Pregel-style orchestration runtime for stateful LLM agents: typed state with reducers, nodes and edges, checkpointed super-steps, durable interrupts for human-in-the-loop, time travel and subgraphs.
  • LangSmith: LangChain’s hosted (or self-hosted Enterprise) platform for tracing LLM apps as run trees, evaluating them on versioned datasets with code, LLM-judge and human evaluators, and deploying LangGraph agents.
  • LlamaIndex: A Python/TypeScript data framework for LLM apps: Documents and Nodes, an ingestion pipeline, index types, retrievers, postprocessors and response synthesizers composed into query engines, plus event-driven Workflows for agents.
  • Mastra: Open-source TypeScript framework for AI agents and applications: agents with a provider/model router, typed suspendable workflows, layered memory, processors as guardrails, scorers for evals, and a local Studio, deployable into existing Node servers.
  • Microsoft Agent Framework: Microsoft’s open-source .NET/Python (Go preview) SDK that succeeds Semantic Kernel and AutoGen: one AIAgent abstraction over any chat client, a layered middleware/context-provider pipeline, typed graph workflows, and Azure Foundry integration.
  • OpenAI Agents SDK: OpenAI’s lightweight, provider-agnostic Python/TS agent framework built on a Runner loop with agents, handoffs, agents-as-tools, guardrails, sessions and built-in tracing.
  • Pydantic AI: A typed Python agent framework from the Pydantic team: Agent[Deps, Output], dependency injection via RunContext, validated structured output with retry, usage limits, capabilities, durable execution engines, and test models.
  • smolagents: Hugging Face’s minimal Python agent library built around code actions: a CodeAgent writes Python that an AST-walking interpreter or a remote sandbox runs, with a ToolCallingAgent for JSON tool calls, any model backend, MCP and Hub tools, and hierarchical managed agents.
  • Vercel AI SDK: Vercel’s TypeScript toolkit for LLM apps: a provider-agnostic Core (generateText/streamText, Output schemas, tools, stopWhen/prepareStep, ToolLoopAgent, WorkflowAgent), UI hooks like useChat over an SSE stream protocol, and harness adapters.

Protocols

  • Agent Authentication and Authorization: How agents act for users without holding their keys: delegated vs agent identities, OAuth 2.1 in MCP (discovery, CIMD/DCR, audience-bound scoped tokens, step-up), token exchange and ID-JAG for enterprise SSO, per-user credential brokers (Arcade, Composio, Vercel Connect, Auth0, AgentCore), least privilege and audit.
  • Agent2Agent (A2A) Protocol: Open protocol for agents from different vendors to discover each other via Agent Cards and delegate stateful, long-running Tasks over JSON-RPC, gRPC or REST; the horizontal complement to MCP.
  • MCP in Practice: How the Model Context Protocol actually works on the wire and in production: primitives, the stateless 2026-07-28 revision, stdio vs Streamable HTTP, OAuth authorization, registries, SDKs and security failure modes.

Security and safety

  • Content moderation: Classifying user and model text or images against a harm taxonomy (hate, sexual, violence, self-harm…) to block, flag or route it: taxonomies, scores and severities, the main classifiers, and how to tune them.
  • Guardrail tooling: The libraries, classifiers and managed services that check LLM inputs, outputs and tool calls at runtime: NeMo Guardrails, Guardrails AI, Llama Guard / Prompt Guard, Bedrock Guardrails, Azure Content Safety, OpenAI moderation.
  • Jailbreaks: Inputs that get a model to produce content its safety training should refuse: why safety training fails, the main attack families (role-play, encodings, adversarial suffixes, many-shot, multi-turn, automated and Best-of-N), defences such as constitutional classifiers, and the over-refusal trade-off.
  • Lethal trifecta: Willison’s rule that an agent with private data, untrusted content and an outbound channel can be made to leak that data, Meta’s Agents Rule of Two that generalises it, and how to apply both when designing agents.
  • LLM Red Teaming: Systematic adversarial testing of LLM applications, manual and automated (garak, PyRIT, promptfoo), scoped and reported against OWASP LLM/Agentic Top 10 and MITRE ATLAS.
  • LLM security risks: The threat model for LLM applications and agents, organised around the OWASP Top 10 for LLM Applications (2026 edition, with the 2025 mapping): what each risk is, how it happens, and the controls that work.
  • OWASP Top 10 for LLM and Agentic Applications: OWASP’s two community risk lists for generative AI: the LLM Top 10 (2023-24, 2025 and 2026 editions) and the Agentic Top 10 (ASI01-ASI10), what each entry means, how the editions differ, and how to use them in a design or code review.
  • PII and Data Privacy: Engineering personal-data protection in LLM apps: where PII flows (prompts, logs, embeddings, caches), detection and redaction with Presidio, zero data retention, training opt-outs, data residency, and GDPR implications.
  • Prompt injection: Attacks that smuggle instructions into an LLM’s context through user input or retrieved content; why it is unsolved at the model level, how exfiltration works, and the defences that actually bound it (spotlighting, Dual LLM, CaMeL, design patterns, Rule of Two, least privilege).

Platforms

  • Amazon Bedrock: AWS’s managed, multi-provider foundation-model service: inference APIs, cross-Region profiles, quotas and capacity tiers, Knowledge Bases, Guardrails, AgentCore, and the data-protection model.
  • Anthropic Messages API: Claude’s stateless generation endpoint: content-block request/response shape, client vs server tools and pause_turn, adaptive thinking with signed blocks, opt-in caching (breakpoints or automatic), Files API, versioning, limits and porting gotchas.
  • Azure AI Foundry: Microsoft’s AI platform (renamed Microsoft Foundry), which includes Azure OpenAI: the Foundry resource model, deployment types and data zones, PTUs and spillover, quota tiers, guardrails, Agent Service and network isolation.
  • Gemini API: Google’s Gemini developer API: the stateful step-based Interactions API vs legacy generateContent, built-in tools (Search grounding, code execution, File Search), thinking levels and thought signatures, implicit vs explicit caching, Files API, tiers and the OpenAI-compatible layer.
  • Google Vertex AI: Google Cloud’s AI platform, folded into Gemini Enterprise Agent Platform in April 2026: Gemini and Model Garden, endpoint types and residency, PayGo tiers and Provisioned Throughput, RAG Engine, Agent Runtime, and VPC-SC.
  • LiteLLM: BerriAI’s open-source Python SDK and self-hosted OpenAI-compatible proxy for 100+ model providers: provider-prefixed model names, a Router with fallbacks and cooldowns, Postgres-backed virtual keys and budgets, callback-based logging, and a 2026 record of supply-chain and proxy CVEs.
  • LLM Gateways: A proxy layer between applications and model providers that gives one (usually OpenAI-compatible) API plus centralised keys, budgets, retries, fallbacks, caching and logging; LiteLLM, OpenRouter, Portkey, Vercel AI Gateway, Cloudflare, Kong and Envoy compared.
  • Model Routing and Fallbacks: Choosing which model or provider serves each request: reliability fallbacks and 429 handling, plus cost-quality routing (cascades, learned routers like RouteLLM, difficulty-based routing) with measured savings and failure modes.
  • OpenAI Responses API: OpenAI’s primary generation endpoint: typed input/output items, optional server-side state, hosted tools, encrypted reasoning, compaction and background mode, and how it differs from Chat Completions.
  • Provisioned Throughput: Reserved LLM inference capacity bought in units (Azure PTUs, Vertex GSUs, Bedrock model units and Reserved tier, OpenAI Scale/Reserved Tier) for a fixed term: how each unit converts tokens through burndown weights, how to size it from peak traffic, quota versus capacity, spillover, and when it beats pay-as-you-go.
  • Rate Limits and Retries: How LLM APIs meter traffic (RPM, TPM, ITPM/OTPM, daily and spend limits, ramp limits), how to classify 429/503/529 failures, and how to build clients that survive them: backoff with jitter, retry budgets, timeouts for long generations, concurrency caps, queueing and service tiers.

Sources