跳到正文- Synthetic Data Generation
- Source: τ²-bench: Evaluating Conversational Agents in a Dual-Control Environment (Barres et al., 2025)
- Source: τ-bench: Tool-Agent-User Interaction Benchmark (Yao et al., 2024)
- Source: Why Do Multi-Agent LLM Systems Fail? (Cemri et al., 2025)
- Source: Who Validates the Validators? (Shankar et al., 2024)
- Source: vCache, verified semantic prompt caching (Schroeder et al., 2025)
- Source: Using LLM-as-a-Judge for evaluation (Hamel Husain)
- Source: UMBRELA: An Open-Source Reproduction of the Bing Relevance Assessor (Upadhyay et al.)
- Source: The TREC-8 Question Answering Track Report (Voorhees)
- Source: The Leaderboard Illusion (Singh et al., 2025)
- Source: The illusion of a perfect metric (Oliva et al., 2025)
- Source: SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (Jimenez et al., 2023)
- Source: SWE-bench evaluation harness documentation
- Source: SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (Li et al., 2026)
- Source: Same fabricated answer, RAGAS scored 0.0 and DeepEval 1.0 (Nick Lamb)
- Source: RouterBench, a benchmark for multi-LLM routing (Hu et al., 2024)
- Source: Reducing latency (Anthropic docs)
- Source: Ragas docs: Testset generation for RAG
- Source: Ragas docs: context precision, context recall, faithfulness, and release notes
- Source: Ragas docs: Answer Relevancy and Noise Sensitivity metrics
- Source: Promptfoo docs: Red team overview and configuration
- Source: Promptfoo docs: CLI, configuration guide, CI/CD and GitHub Action
- Source: Promptfoo docs: Assertions and metrics
- Source: Pervasive Annotation Errors Break Text-to-SQL Benchmarks (Jin et al., 2026)
- Source: OpenTelemetry semantic-conventions CHANGELOG (GenAI entries)
- Source: OpenTelemetry semantic conventions for generative AI
- Source: OpenTelemetry GenAI conventions: events, tool, agent, MCP and registry details
- Source: OpenLLMetry (Traceloop) README, conventions and ServiceNow announcement
- Source: openai/evals (GitHub)
- Source: OpenAI wants to retire SWE-bench Verified (The Decoder, 2026)
- Source: OpenAI acquires Promptfoo to secure its AI agents (TechCrunch, 2026)
- Source: Notebook: LLM Evaluation & Observability
- Source: Non-Determinism of "Deterministic" LLM Settings (Atil et al., 2024)
- Source: MMLU-Pro (Wang et al., 2024)
- Source: Migrate a Promptfoo eval suite to Langfuse (Langfuse guide)
- Source: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (Becker et al., 2025)
- Source: Measuring Massive Multitask Language Understanding (Hendrycks et al., 2020)
- Source: Mastra docs: Evals (scorers) overview
- Source: LLM Evaluators Recognize and Favor Their Own Generations (Panickssery et al., 2024)
- Source: LlamaIndex retrieval evaluation metrics (source code)
- Source: LiveBench: A Challenging, Contamination-Limited LLM Benchmark (White et al., 2024)
- Source: Length-Controlled AlpacaEval (Dubois et al., 2024)
- Source: Latency optimization (OpenAI guide)
- Source: Large Language Models Can Accurately Predict Searcher Preferences (Thomas et al.)
- Source: Large Language Models are not Fair Evaluators (Wang et al., 2023)
- Source: LangSmith prompt engineering and prompt management docs
- Source: LangSmith observability documentation
- Source: LangSmith evaluation concepts and SDK docs
- Source: Langfuse prompt management documentation
- Source: Langfuse pricing page and v4 changelog
- Source: Langfuse observability documentation
- Source: Langfuse evaluation, experiments and CI/CD documentation
- Source: Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (Ye et al., 2024)
- Source: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al.)
- Source: Introduction to Information Retrieval, ch. 8: Evaluation in IR (Manning, Raghavan, Schütze)
- Source: Introducing SWE-bench Verified (OpenAI, 2024)
- Source: Inspect documentation (UK AI Security Institute)
- Source: Humanity's Last Exam (Phan et al., 2025)
- Source: Holistic Agent Leaderboard (Kapoor et al., 2025)
- Source: GroUSE, a benchmark to evaluate evaluators in grounded QA (Muller et al., 2024)
- Source: GPTCache, an open-source semantic cache for LLM applications (Fu & Feng, 2023)
- Source: GPT Semantic Cache (Regmi & Pun, 2024)
- Source: GPQA: A Graduate-Level Google-Proof Q&A Benchmark (Rein et al., 2023)
- Source: G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (Liu et al., 2023)
- Source: Evaluation best practices (OpenAI API docs)
- Source: Evaluating Large Language Models Trained on Code (Chen et al., 2021)
- Source: Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? (Gloaguen et al., 2026)
- Source: Elastic License 2.0 (ELv2) text
- Source: Detecting hallucinations in large language models using semantic entropy (Nature, 2024)
- Source: Demystifying evals for AI agents (Anthropic)
- Source: Define success criteria and build evaluations (Claude docs)
- Source: DeepEval docs: Answer Relevancy and Contextual Recall metrics
- Source: DeepEval docs (G-Eval, DAG, faithfulness, contextual precision, evaluation flow) and Confident AI pricing
- Source: Datadog LLM Observability: OpenTelemetry GenAI ingestion
- Source: Cumulated Gain-Based Evaluation of IR Techniques (Järvelin and Kekäläinen)
- Source: Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (Chiang et al., 2024)
- Source: Build LLM Apps tutorial (Notebook)
- Source: Braintrust platform docs: experiments, scorers, datasets, online scoring, architecture, pricing
- Source: Braintrust docs: Prompts
- Source: Braintrust docs: Eval SDK quickstart
- Source: Braintrust autoevals (GitHub)
- Source: BFCL V3: Multi-Turn and Multi-Step Function Calling (Berkeley)
- Source: BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of IR Models (Thakur et al.)
- Source: Arize Phoenix docs: evals, experiments, phoenix-otel, self-hosting, and Arize pricing
- Source: Arize Phoenix and OpenInference documentation
- Source: Are We Done with MMLU? (Gema et al., 2024)
- Source: Are Emergent Abilities of Large Language Models a Mirage? (Schaeffer et al., 2023)
- Source: ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems (Chollet et al., 2025)
- Source: ARC-AGI series and ARC-AGI-3 (ARC Prize Foundation)
- Source: AI Evals: Everything You Need to Know (Husain and Shankar)
- Source: AI Agents That Matter (Kapoor et al., 2024)
- Source: agentevals (LangChain)
- Source: ADK documentation: Why evaluate agents
- Source: Adding Error Bars to Evals (Miller, 2024)
- Source: A Field Guide to Rapidly Improving AI Products (Hamel Husain)
- Source: A Careful Examination of LLM Performance on Grade School Arithmetic (Zhang et al., 2024)
- Retrieval Metrics
- RAGAS and DeepEval
- promptfoo
- Prompt Management
- OpenTelemetry GenAI Semantic Conventions
- LLM-as-Judge
- LLM Red Teaming
- LLM Observability
- LLM Non-Determinism
- LLM Evaluation
- LangSmith
- Langfuse
- Harness Engineering
- Error Analysis
- Cost and Latency Optimization
- Braintrust
- Benchmarks
- Arize Phoenix
- Agent Evaluation