Prompt Management
Treating prompts as versioned, deployable artifacts: registries with immutable versions and movable labels, caching and fallbacks, trace linkage, and the eval-gated promotion loop.
Contents
Prompt management is the practice of treating prompts as versioned, deployable artifacts rather than string literals scattered through code. That means a registry with an immutable version history, named pointers (labels, tags or environments) that decide which version each environment serves, runtime fetching with caching, a record on every trace of which version produced it, and promotion gated by evals. It matters because a prompt tweak or a model upgrade can quietly break cases that used to work (notebook). Without versioning you cannot tell which prompt caused a regression or roll it back. The craft of writing the prompt itself is prompt engineering. This page covers its lifecycle.
Core mechanics (common to all registries)
- Immutable versions. Each save creates a new version that never changes: numbered in Langfuse, a commit hash in LangSmith, a resolved version ID in Braintrust (which its Ruby SDK docs call a transaction ID) (Langfuse, LangSmith, Braintrust).
- Movable pointers. Code references a pointer, not a version. Langfuse serves
productionby default and maintainslatestautomatically. LangSmith reserves the tagsstagingandproductionfor its Environments feature. Braintrust loads by environment. Deploying or rolling back means moving a pointer, with no code change. - Template + variables. Templates are
{{var}}(Langfuse, Braintrust) or f-string/mustache (LangSmith). They come as text or chat (message list) types. Langfuse can’t change a prompt’s type after creation, and LangSmith recommends chat. - Config travels with the prompt. Model, temperature, a
response_formatschema andtoolsare versioned together with the template (Langfuseconfig, LangSmithinclude_model, Braintrustbuild()returning messages + model + params). Synthesis: this matters because a prompt is tuned for one model. Promoting new text while the model parameters stay in code silently mixes versions. - Composition. Langfuse prompt references (
@@@langfusePrompt:name=X|label=production@@@, text prompts only) and message placeholders for chat history. - Trace linkage. Every LLM span records which prompt version produced it. Langfuse links generations and breaks latency, tokens, cost and scores down per version. Braintrust writes the prompt ID and version to
metadata.prompton wrapped-client spans. The OTel GenAI spec definesgen_ai.prompt.nameandgen_ai.prompt.versionfor any backend (OTel GenAI).
Runtime fetching: caching and availability
A registry sits on the request path, so how its SDK caches determines both latency and outage behaviour:
| Langfuse | LangSmith | Braintrust | |
|---|---|---|---|
| Default cache | in-memory, TTL 60 s, stale-while-revalidate | process-wide in-memory, TTL 300 s, refresh every 60 s, max 100 prompts | fetch every call; in-memory + disk cache used only on transient errors |
| Disable / bypass | cache_ttl_seconds=0 | skip_cache=True, disable_prompt_cache=True | n/a |
| Outage behaviour | serve stale cache; optional fallback prompt if cache empty | serve cache; offline mode via dump()/load() | return last cached copy on network/429/5xx, except loads pinned to a version or environment, which raise |
| Uncached fetch latency | ~39 ms mean, ~69 ms p99 (vendor benchmark) | not stated | not stated |
Sources: Langfuse, LangSmith, Braintrust.
Consequences:
- Propagation delay: after you move a label, instances keep serving the old version until their TTL expires. The first traces after a change may show the old version.
- Cold start plus outage = failure. Prefetch at startup and ship a fallback prompt. In Langfuse a call served from the fallback is not linked to any prompt version, so it shows up as unattributed.
- Missing label = hard error. Langfuse returns 404 with no fallback to
production.
Prompts in the registry vs prompts in Git
| Registry (Langfuse, LangSmith, Braintrust, Phoenix) | Prompts as code (files in repo, e.g. promptfoo file://) | |
|---|---|---|
| Who edits | engineers and non-engineers in a UI | engineers via PR |
| Deploy | move a label; live within the cache TTL | merge + deploy |
| Review/gate | webhooks → CI; protected labels | PR review + CI eval as a required check |
| Reproducibility | version ID on each trace | Git SHA on each trace |
| Risk | runtime dependency; edits bypass code review | slower iteration; non-engineers blocked |
Synthesis: many teams use both. The prompt is authored in Git and pushed to the registry by CI after the evals pass, so the registry serves as a deployment channel rather than the source of truth. The other direction also works: Langfuse webhooks fire on version creation and on label moves, with an HMAC-signed payload that includes the template and config. A handler can then start an eval run before anyone promotes the version (Langfuse). Protected labels, which need Langfuse Enterprise or Pro with the Teams add-on, stop non-admins from moving production (Langfuse). Arize Phoenix also ships prompt management alongside its playground (Phoenix). The two styles can coexist: promptfoo can evaluate Langfuse-managed prompts through langfuse:// references (migration guide). For code-only prompts, Braintrust recommends logging your own name, version or Git SHA in span metadata (Braintrust).
The eval-gated promotion loop
flowchart TD
E["Edit prompt → new immutable version (label: latest)"] --> O["Offline eval: run app over versioned dataset"]
O --> G{"Gate: no approved case regresses AND aggregate ≥ threshold"}
G -->|"fail"| E
G -->|"pass"| S["Move label: staging → canary / A-B label"]
S --> P["Move label: production"]
P --> M["Online: traces tagged with prompt version, sampled LLM-judge scores, user feedback"]
M -->|"regression"| RB["Roll back: move production label to previous version"]
M -->|"new failure cases"| D["Add to dataset (regression suite)"]
D --> OSynthesis: this diagram combines the sources below.
- Dataset first. Start with 20-50 cases drawn from real failures, covering happy paths, hard cases and edge cases. Never let eval cases leak into the few-shot examples (notebook, Anthropic). OpenAI lists vibe-based evaluation and adding evals only after shipping as anti-patterns (OpenAI).
- Change one variable at a time (prompt, or model, or parameters) and re-score against the baseline (notebook).
- Gate in CI. promptfoo exits with code 100 when tests fail or the pass rate is below
PROMPTFOO_PASS_RATE_THRESHOLD, and its GitHub Action runs on PRs that touch prompt files (promptfoo). Langfuse’s experiment Action fails the PR on aRegressionError. Its docs warn that an aggregate threshold can hide a newly failing critical case. They recommend a reviewed baseline where no approved passing case may regress, treating missing results as failures, and never generating the baseline from the candidate itself (Langfuse evals). Pin the dataset version, for example Langfuse’sdataset_version, so the gate is reproducible. - Capability vs regression suites. Regression suites should pass at close to 100%. Capability evals start low, and tasks that become saturated graduate into the regression suite (Anthropic).
- Online. Compare per-version metrics after promotion (latency, tokens, cost, median score in Langfuse), and use custom labels to split traffic in A/B tests (Langfuse).
Failure modes
- Version skew: the prompt is promoted but its config (model, schema, tools) still lives in code, or the reverse.
- Silent edits: a UI edit moves
productionwith no eval run. Fix it with protected labels plus webhook-triggered evals. - Untraceable outputs: spans lack a prompt version (unlinked calls, fallback served), so regressions can’t be attributed.
- Cache surprises: tests read a stale cached version. Disable caching in test and dev environments.
- Overfitting the gate: iterating against the same small dataset until it passes. Synthesis: keep a held-out split and keep feeding in fresh production failures.
- Leaky composition: a referenced sub-prompt that follows
label=productionchanges the parent’s behaviour without a new parent version. Pin references byversionfor audited flows (Synthesis, from the reference semantics).
When to use / when not
- Use a registry when non-engineers iterate on prompts, prompts change more often than code deploys, you need instant rollback, or many services share prompts.
- Keep prompts in Git (and still gate them with promptfoo or another eval) for small teams, tightly coupled prompt+code changes such as tool schemas in function calling, or regulated settings where every change must pass code review. Record the Git SHA on traces.
- For automatic prompt optimisation instead of hand edits, see DSPy. Tools: Langfuse, LangSmith, promptfoo.
Sources
- Langfuse prompt management documentation
- LangSmith prompt engineering and prompt management docs
- Braintrust docs: Prompts
- Langfuse evaluation, experiments and CI/CD documentation
- Promptfoo docs: CLI, configuration guide, CI/CD and GitHub Action
- Migrate a Promptfoo eval suite to Langfuse
- OpenTelemetry semantic conventions for generative AI
- Notebook: LLM Evaluation & Observability
- Demystifying evals for AI agents (Anthropic)
- Evaluation best practices (OpenAI)
- Arize Phoenix and OpenInference documentation