Source: Promptfoo docs: CLI, configuration guide, CI/CD and GitHub Action
Promptfoo reference for promptfooconfig.yaml structure, eval CLI flags, exit codes and pass-rate thresholds, caching, and running evals and red teams in CI.
Contents
- What: official tool documentation and repository README
- Publisher: Promptfoo (now part of OpenAI, per its README)
- Date: accessed 2026-11-14
- Original: Intro, Configuration guide, Command line, CI/CD, GitHub Action, GitHub repo
Summary
Promptfoo is an open-source (MIT) CLI and library for evaluating and red-teaming LLM apps. It runs locally, calling model providers directly, and reads a declarative YAML config. The CLI exits non-zero when tests fail, which is how it gates CI; a GitHub Action posts before/after comparisons on pull requests that touch prompt files.
Key claims
- Positioning: evals of prompts, models and RAG pipelines, plus red teaming; caching, concurrency and live reload; matrix views, a web viewer and sharing. Evals run on the user’s machine and talk directly to the LLM.
- Repository (accessed 2026-11-14): MIT licensed, about 27k GitHub stars; README states Promptfoo is now part of OpenAI and remains open source. Install via
npx promptfoo@latest,npm install -g promptfoo,brew install promptfooorpip install promptfoo. The GitHub Action needs Node.js 22.22.0+. - Config keys:
prompts(inline orfile://),providers(id,label,configsuch as temperature),tests(inline, globs, CSV, JSONL, YAML/JSON, JS/TS),defaultTest(sharedassert,vars,threshold,options),assertionTemplatesreused via$ref. Multiple-cconfigs can be combined. - Vars can come from files or scripts (JS function or Python
get_var); Nunjucks templating in prompts and vars. - Transforms: provider
transformResponse, testoptions.transform(before assertions),options.transformVars,contextTransformfor RAG assertions.defaultTest.options.providersets the grading (judge) model separately from the models under test. promptfoo evalflags:--repeat N,-j/--max-concurrency,--delay,--filter-failing <path|evalId>,--filter-pattern,--no-cache,-o(csv/json/yaml/html/junit.xml),--share/--no-share,--grader,-w/--watch,--table.promptfoo retry <evalId>re-runs only ERROR results.- Exit code 100 when any test fails or the pass rate is below
PROMPTFOO_PASS_RATE_THRESHOLD(percentage, default 100); exit 1 for other errors;PROMPTFOO_FAILED_TEST_EXIT_CODEoverrides 100. - Disk cache of LLM requests/outputs (
PROMPTFOO_CACHE_PATH,PROMPTFOO_CACHE_TTL) can be persisted between CI runs to cut cost and time. - CI privacy knobs:
PROMPTFOO_STRIP_RESPONSE_OUTPUT,PROMPTFOO_STRIP_TEST_VARS;--tag key=valueattaches pipeline metadata. promptfoo/promptfoo-action@v1: triggered bypull_requestwith a path filter on prompt files; inputsgithub-token,prompts(globs),config, optionalopenai-api-keyandcache-path; posts a PR comment linking to a viewer that compares before and after.- Red teaming in CI: scheduled jobs run
redteam generatewith chosen plugins and strategies, thenredteam run.