POC to Production

Taking an LLM proof of concept to a dependable production system: why most pilots stall, the stage gates (evals, guardrails, integration, rollout), human-in-the-loop design, change management and the post-launch data flywheel.

Contents

POC to production is the path from a working prototype of an LLM feature (a proof of concept, or POC) to a system that real users depend on, at agreed quality, cost, latency and risk. LLM prototypes are cheap and impressive. Making them dependable is the hard part. The practitioners behind Applied LLMs call this the gap between “0 to 1” demos and “1 to N” products (Applied LLMs). Closing that gap is the core job of a forward deployed engineer.

The pilot-to-production gap

  • Gartner forecast (July 2024) that at least 30% of GenAI projects would be abandoned after POC by end of 2025. The causes it named were poor data quality, inadequate risk controls, escalating costs and unclear business value (Gartner 2024). Its June 2025 forecast puts agentic AI cancellations above 40% by end of 2027 (Gartner 2025).
  • MIT NANDA (2025, preliminary, small self-reported sample) found that only about 5% of enterprise pilots produce rapid revenue impact. It blames a learning gap: tools that do not integrate with workflows, retain feedback or adapt to context. Model quality is not the main problem (MIT NANDA).

Synthesis: these numbers come from forecasts and small samples. The causes they name are consistent with each other: no metric, no integration, no risk controls, no cost model. Each maps to a stage gate below.

Why demos lie

Synthesis, grounded in the sources: a POC usually runs on a curated handful of inputs, often anonymised or offline data. That is how OpenAI describes typical solutions-architect POCs, as opposed to FDE work on customer infrastructure (Pragmatic Engineer). Production adds:

  • Distribution shift. Real inputs include edge cases nobody imagined. Failure modes are unpredictable, so review real traffic daily (Applied LLMs).
  • Compounding unreliability. At 75% per-trial success, the chance that 3 of 3 trials succeed (pass^3) is only about 42% (Anthropic evals). Users see the pass^k behaviour, not the best-case demo.
  • Persistent hallucination. Applied LLMs puts the baseline at roughly 5-10%, which needs downstream checks (Applied LLMs; see hallucination).
  • Integration, permissions, cost at volume, security and ownership.

Stage gates

flowchart LR
  A["Discovery: metric + baseline"] --> B["POC: best model, real data, eval baseline"]
  B -->|"meets threshold?"| C["Hardening: guardrails, integration, cost/latency"]
  B -->|"no"| X["Stop or rescope"]
  C --> D["Pilot: limited users, human review"]
  D -->|"business metric moves?"| E["Staged rollout"]
  D -->|"no"| X
  E --> F["Operate: monitor, feedback to evals"]
  F -->|"new failures"| B

Synthesis: the gate structure combines OpenAI’s scope-validate-deliver phases, its incremental agent guidance and Applied LLMs’ iteration loop.

1. POC (validate feasibility). Prototype with the most capable model to set an eval baseline. Hit the accuracy target first, then swap in smaller models for cost and latency (OpenAI agents guide; see model selection). Start with prompting and retrieval, and only fine-tune once those are proven insufficient (Applied LLMs). Build the smallest end-to-end slice on real systems. OpenAI FDEs validate with evals before they commit to delivery (Pragmatic Engineer). Exit criterion: the eval score meets the threshold agreed in discovery.

2. Hardening (make it dependable).

3. Pilot (validate value). Put the system in front of a limited group of users, measure the business metric against the baseline, and keep humans reviewing outputs. Human intervention matters most early in deployment, when it surfaces failures and builds the eval cycle (OpenAI agents guide). Indeed tested the GPT version against the old engine before scaling (OpenAI, Enterprise).

4. Rollout and operate. Expand in stages. Production logs feed new eval cases and fixes, which is the data flywheel. LLMOps exists to shorten the loop from production back to development (Applied LLMs). Klarna reached two-thirds of service chats through continuous testing and refinement, not a single launch (OpenAI, Enterprise).

Human-in-the-loop design and change management

  • Augment first. Design the UX so users can accept, edit or reject outputs. This builds trust and generates labelled feedback. Treat the LLM as support for human workflows, not a replacement (Applied LLMs).
  • Start narrow. Narrow products that are honest about their limits earn trust faster than generic ones (Applied LLMs).
  • Involve the people closest to the work, plus control functions. BBVA brought Legal, Compliance and IT Security in from the start and let experts build their own tools (OpenAI, Enterprise). Line managers, not only central AI labs, drive adoption (MIT NANDA). See AI governance and compliance.
  • Sponsor visibly. Leadership-led adoption and early quick wins build momentum (OpenAI, use cases).
  • Expect years, not weeks. Neural-network driving was demoed in 1988, and commercial driverless service arrived decades later (Applied LLMs).

Synthesis: change management for LLM systems differs from classic software rollouts in one way. Output quality is probabilistic, so users need a calibrated sense of when to trust the system. Ship with visible uncertainty and an escalation path, and publish the eval metrics to stakeholders.

Production-readiness checklist

Synthesis, consolidating the sources above:

AreaReady when
QualityEval suite of real cases passes the agreed threshold; regression evals run in CI
Reliabilitypass^k measured on critical tasks; fallbacks and retry limits defined
SafetyGuardrails, injection defences, human approval on irreversible actions
Data & accessLeast-privilege credentials; PII handling approved
Cost & latencyCost per task and p95 latency within budget at forecast volume
ObservabilityTraces, feedback capture, alerting on quality drift
OwnershipA customer-side owner exists; runbooks; model-upgrade plan
ValueBusiness metric measured against the baseline in the pilot

Failure modes

  • Pilot purgatory. Gartner names unclear business value among the main reasons POCs are abandoned (Gartner 2024). Synthesis: a POC with no agreed business metric and no customer-side owner leaves nobody able to approve production.
  • Generic tool, no workflow integration. This is the learning gap (MIT NANDA).
  • Overbuilding. Going multi-agent or autonomous before a single agent works (OpenAI agents guide), or training models before product-market fit (Applied LLMs).
  • Bespoke forever. For vendors, custom deployments that are never turned into reusable templates. Time-box them and convert custom code to templates (a16z).
  • Model-centric thinking. Models change fast. The durable assets are evals, guardrails, caching and the data flywheel (Applied LLMs).

Sources

Suggest changes →