Web Search APIs for Agents

How agents get live web access: provider-native search tools, LLM-oriented search APIs (Tavily, Exa, Brave, Perplexity, SerpApi), page extraction (Firecrawl, Jina Reader), citation grounding, and the cost, freshness and security trade-offs.

Contents

Web search APIs for agents are the tools that give a model live access to the web: a search step that turns a query into ranked URLs (often with extracted text), and a fetch/extract step that turns a chosen URL into clean, token-efficient content. They are the standard fix for a model’s training cutoff and a main source of grounding against hallucination. They come in two shapes. Provider-native tools run inside the model API’s server-side loop and return citations automatically. Third-party APIs you wrap as a function-calling tool yourself, which gives you control over the index, filters, cost and which model you use.

The two halves: search and fetch

Synthesis: almost every web-enabled agent combines two operations, and the vendors differ mainly in how much of each they bundle.

OperationInput → outputExamples
Searchquery → ranked results (title, URL, snippet or chunks)Claude web_search, OpenAI web_search, Gemini google_search, Tavily, Exa, Brave, Perplexity Search, SerpApi
Fetch / extractURL → main content as markdown/text/JSONClaude web_fetch, OpenAI open_page/find_in_page actions, Gemini url_context, Firecrawl, Jina Reader, Tavily Extract

Anthropic documents the combined flow directly: with both tools enabled, Claude searches, picks promising results, fetches full content, then answers with citations (web fetch docs). SerpApi returns only result-page snippets, so a fetch step is always needed (SerpApi). Tavily and Exa return extracted content in the search call itself (Tavily, Exa).

flowchart LR
  Q["User question"] --> M["Model decides to search"]
  M --> S["Search API: query to ranked URLs + snippets"]
  S --> F{"Snippets enough?"}
  F -- "no" --> X["Fetch / extract: URL to markdown"]
  F -- "yes" --> C["Filter / rerank into context"]
  X --> C
  C --> A["Answer with citations (URL + span)"]
  A --> M

Synthesis: the loop arrow is the important part. Agentic search means repeated search and fetch rounds inside one turn, not a single retrieval as in classic RAG.

Provider-native search tools (as of 2026-11)

Anthropic web_searchOpenAI web_search (Responses)Gemini google_search
Who runs the loopAPI server (server_tool_use), may pause_turnAPI server; reasoning models add open_page, find_in_pageAPI server
Domain controlallowed_domains or blocked_domains (not both)filters.allowed/blocked_domains, up to 100 eachnot in the guide
Volume controlmax_uses per turnsearch_context_size low/medium/high, return_token_budgetnone documented
Freshness controlweb_fetch_20260309+ use_cache: falseexternal_web_access: false = cache-onlyn/a
Citationsalways on; cited_text up to 150 chars, not billed as tokensurl_citation annotations with char span; sources lists all URLs consultedurl_citation with start_index/end_index
Price$10 / 1k searches + tokens; failed searches freeper-call tool fee + tokens (see pricing page)Gemini 3+: 5k free/month then $14 / 1k queries; Gemini 2.5 and older billed per prompt

Sources: Anthropic search, OpenAI, Gemini.

Details that matter in production:

  • Context filtering. Basic search dumps every result into context. Claude’s web_search_20260209+ and web_fetch_20260209+ let the model write code that filters results before they reach the context (“dynamic filtering”). It runs search from inside code execution, which the API provisions automatically; it is on by default (allowed_callers defaults to code execution) and is turned off with allowed_callers: ["direct"]. It needs Claude 4.6+ models, is not ZDR/HIPAA eligible, and is unavailable on Google-managed Vertex and Azure-hosted Foundry deployments, which offer only web_search_20250305. OpenAI caps search context at 128k tokens whatever the model’s window. (Anthropic, OpenAI) This is context engineering applied to the web.
  • Multi-turn state. Claude results carry encrypted_content and citations carry encrypted_index. Both must be sent back unchanged, or the next request fails with a 400. Search results also count as input tokens on every later turn, so they are worth prompt caching. (Anthropic)
  • Errors are in-band. Claude returns HTTP 200 with web_search_tool_result_error codes (max_uses_exceeded, too_many_requests, query_too_long…), so your loop must inspect content, not status codes. (Anthropic)
  • Display obligations. All three vendors require citations or search suggestions to be shown to end users. OpenAI asks for clickable inline links; Google’s search-suggestion HTML is governed by its terms. (OpenAI, Gemini, Anthropic)
  • Model churn. OpenAI shut down the gpt-4o-search-preview-2025-03-11 and gpt-4o-mini-search-preview-2025-03-11 snapshots on 2026-07-23 (named replacement gpt-5.6-terra). Chat Completions now offers only gpt-5-search-api, which always searches and has no domain filters. (deprecations, OpenAI)

Third-party search APIs

APIIndex / mechanismReturnsKey knobsPrice (2026-11)
Tavilyaggregator tuned for LLM agentsresults with relevance score + 1-3 query-relevant chunks per source; optional raw content and answersearch_depth ultra-fast/fast/basic/advanced, topic, time_range, include_domains (300)1 credit (advanced 2); $0.008/credit PAYG, 1k free/month
Exaown index; embedding-style retrieval (docs show a neural cost line)results + highlights, full text, per-result summary, or schema answertype lite/auto/deep-lite/deep, category, maxAgeHours, 1,200-domain filtersper 1k requests (up to 10 results): lite $4, auto $7, deep-lite/deep $12; contents $1/1k pages per type
Braveown independent index, no Google/Bing fallbackSERP JSON; LLM Context endpoint with extracted content (~1k-33k tokens)vertical endpoints, Goggles$5/1k searches at 50 QPS; Answers $4/1k + $5/M tokens; $5 free credit/month
Perplexityown search stackSearch API: ranked results; sonar: grounded answersearch_type: fast; Agent API effort presetsSearch $5/1k ($1/1k fast); sonar $0.25/$2.50 per M tokens
SerpApiscrapes live Google SERPsparsed blocks: organic, knowledge graph, local, related questionslocation, gl, hl, device, no_cacheplans (not on the reference page); cached hits free

Sources: Tavily, Exa, Brave, Perplexity, SerpApi.

How to tell them apart. Synthesis:

  • Google-parity vs own index. SerpApi shows you exactly what Google shows, including local and shopping blocks. That is useful for SEO, local or commerce agents, but it is snippet-only and depends on scraping. Brave and Exa have independent indexes, which means different recall and a different exposure to SEO spam. Brave also sells storage/training rights only on custom plans, which matters if you want to cache results into your own vector database (Brave).
  • Keyword vs semantic queries. Exa expects descriptive natural-language queries (“describe subject, source type, period”) (Exa). SerpApi supports Google operators like site:. Agents write better queries once the tool description says which style the backend wants (see query transformation).
  • Pre-chunked content saves tokens. Tavily’s chunks and Exa’s highlights are already a retrieval-and-reranking step. Asking for raw full text instead can multiply context cost by 10x or more.
  • Answer endpoints (Tavily include_answer, Exa deep/outputSchema, Brave Answers, Perplexity sonar) put a second LLM between the web and your model. That is fast, but you lose control of synthesis and citation granularity.

Content extraction and crawling

ToolModelJS renderingFreshness / cacheCost
Claude web_fetchserver tool; only URLs already in contextno (use browser tool)cached by default; use_cache: falsetokens only
Firecrawlscrape/crawl/map/search/interact/agent; self-hostable OSSyes, plus proxies and anti-botmaxAge default 2 days (up to 5x faster); 0 forces fresh1 credit/page; JSON +4; PDF 1/page
Jina Readerprefix r.jina.ai/<url>; s.jina.ai searchheadless by default, X-Engine: direct fasterX-No-Cache, X-Cache-Tolerancebilled per output token; 10M free

Sources: web fetch, Firecrawl, Jina.

Page sizes drive cost. Anthropic estimates about 2.5k tokens for an average page, 25k for a large docs page and 125k for a research PDF, and offers max_content_tokens to truncate (web fetch). Schema-driven extraction (Firecrawl JSON, ReaderLM-v2 with a schema header) turns pages into structured outputs before they reach the main model (Firecrawl, Jina). Crawling whole sites (Firecrawl Crawl/Map, Tavily Crawl) is ingestion for an offline index, not live search; see document processing.

Citation grounding

Grounding means every claim in the answer points to a span in a retrieved source. Native tools return structured citations: Claude gives a URL, title and cited_text; OpenAI and Gemini give a URL, title and character span in the answer (Anthropic, OpenAI, Gemini). With third-party APIs you build this yourself: number the sources in context, ask for [n] markers or a structured output of claim-source pairs, then check that each cited URL was actually retrieved. OpenAI’s sources list is usually longer than the citations, which shows the gap between what was read and what was credited (OpenAI). Anthropic’s research system runs a dedicated citation pass after writing (multi-agent research); see deep research agents.

Cost and freshness trade-offs

  • Two meters. You pay per search (about $1-14 per 1k across vendors as of 2026-11, from Perplexity fast search to Gemini grounding) plus tokens for everything placed in context. On native tools the token bill usually dominates, because results stay in the history on every later turn (Anthropic, Perplexity, Gemini, Exa).
  • Freshness is a cache setting. Exa maxAgeHours, Firecrawl maxAge, Jina cache tolerance, Claude use_cache, OpenAI external_web_access and SerpApi’s 1-hour cache all trade latency and price against staleness. Forcing live fetches is slower and fails more often (Firecrawl, Exa). Note that maxAgeHours controls content age, not publication date; use date filters for recency.
  • Latency tiers. Exa lite takes about 300 ms (cache only), auto about 1 s and deep 4-15 s (Exa). Perplexity’s effort presets range from 1 step to 300 (Perplexity). Voice and chat UX need the cheap end; see cost and latency optimization.
  • Budget caps. Use max_uses (Claude), search_context_size (OpenAI) and max_results / chunks_per_source (Tavily) to bound runaway loops.

Failure modes

  • Prompt injection from pages. Fetched content is untrusted input that can redirect the agent or exfiltrate data. Claude’s fetch tool refuses URLs the model invented or that contain credentials, but the docs still warn of residual risk when untrusted content meets sensitive data. OpenAI advises running public web search and private MCP lookups in separate requests. (web fetch, OpenAI) See prompt injection.
  • Low-quality sources. Agents favoured SEO content farms over authoritative sources until source-quality heuristics were added (multi-agent research). Domain allow-lists are the blunt fix.
  • JS-heavy or blocked pages. Basic fetchers do not render JavaScript. robots.txt and private addresses are refused (web fetch). Fall back to Firecrawl or Jina rendering, or to computer use.
  • Context flooding. Raw results are mostly irrelevant, which is why dynamic filtering and pre-chunked results exist (Anthropic). See long-context degradation.
  • Citation drift. A citation can point to a real page that does not support the claim. Check claims against cited_text or spans, using an LLM-as-judge if needed.

When to use / when not

  • Native tool: the fastest path when you are already on one provider and want citations, ZDR or compliance handling and less plumbing. You accept the vendor’s index and opaque ranking.
  • Third-party API: when you need model portability, a specific index (Google SERP parity, an independent index, semantic search), fine control over chunks and filters, or one search layer shared by several models (often exposed through MCP).
  • Web search as RAG fallback: Corrective RAG grades local retrieval and falls back to web search only when it fails; the owner’s experiment wires this with a stub search (CRAG paper, notebook). See agentic RAG.
  • Not at all: for stable knowledge, closed-domain enterprise questions answerable from internal data, or where data-egress rules forbid sending queries to third parties. Search queries themselves can leak sensitive intent; see PII and data privacy.

Sources

Suggest changes →