AI Use-Case Discovery
How to find, qualify, scope and prioritise LLM use cases in an organisation: where to look, fit tests for LLMs and agents, impact/effort scoring, success metrics and build-vs-buy.
Contents
AI use-case discovery is the front end of an enterprise LLM engagement. You find the workflows where a language model can create measurable value, test whether an LLM (or an agent) is the right tool for each, define success in measurable terms, and rank the candidates by value and effort. The output is a scoped problem with an agreed metric and a small eval set, ready for a proof of concept. It is the first phase of the forward deployed engineer’s work. OpenAI’s FDEs map processes onsite, prototype and prioritise, then validate the scope with evals before they build (Pragmatic Engineer).
Why it matters
Synthesis: many failed AI projects fail at the selection stage rather than because of the model. Gartner forecasts that over 40% of agentic AI projects will be canceled by end of 2027 because of cost, unclear value and weak risk controls, and advises pursuing agents only where value is clear (Gartner 2025). MIT NANDA found that over half of GenAI budgets go to sales and marketing, while the largest returns come from back-office automation. Startups that target one pain point do well (MIT NANDA).
Step 1: where to look
OpenAI’s guide, drawn from 300 implementations and 600+ use cases, points to three kinds of friction (OpenAI, use cases):
- Repetitive, low-value tasks, such as summarising, report generation and answering the same questions. Ask people for their “anti-to-do list”.
- Skill bottlenecks, where work waits for another team’s expertise (data analysis, code, design).
- Navigating ambiguity, where people get stuck starting open-ended work.
It then classifies use cases into six primitives: content creation, research, coding, data analysis, ideation/strategy and automation. The advanced step is workflow mapping: break a department’s multi-step workflow into tasks and map each task to a primitive.
Discovery techniques in the sources include onsite process mapping (Pragmatic Engineer), workshops and hackathons, and expert-led bottom-up discovery. BBVA rolled out ChatGPT Enterprise and its employees built 2,900+ custom GPTs in five months (OpenAI, Enterprise).
Step 2: does it actually need an LLM, or an agent?
Apply fit tests before scoring. Start with the simplest solution. Optimised single LLM calls with retrieval and examples are often enough (Anthropic, effective agents). OpenAI says agents fit workflows that have resisted deterministic automation (OpenAI, agents guide):
| Signal | Example | Implication |
|---|---|---|
| Complex judgement with exceptions | Refund approval | Agent candidate |
| Rules that are hard to maintain | Vendor security review | Agent candidate |
| Heavy unstructured data | Insurance claim processing | LLM / document processing |
| Clear, stable rules on structured data | Threshold-based alerts | Deterministic code; no LLM |
Synthesis: add four questions that sink projects later. Data access: can the system legally and technically reach the inputs? Error tolerance: what does a wrong answer cost, and who catches it? See hallucination. Ground truth: can experts agree on a correct output? Eval tasks must be unambiguous enough that two experts reach the same verdict (Anthropic, demystifying evals). Volume x value: is the task frequent or costly enough to matter?
Step 3: prioritise
OpenAI’s customer teams use an impact/effort quadrant (OpenAI, use cases):
| Low effort | High effort | |
|---|---|---|
| High value | High ROI focus | Scope and prioritize |
| Low value | Self-service | Deprioritize |
- High ROI focus (high value, low effort): start here to build momentum.
- Scope and prioritize (high value, high effort): often transformational. Indeed’s job-match explanations took months of iteration and lifted applications started by 20%.
- Self-service: individual assistants.
- Deprioritize: for example, rebuilding something an existing tool already does.
Re-score each quarter, because model progress turns high-effort ideas into low-effort ones. The guide also stresses leadership sponsorship and starting with quick wins rather than impressive complex cases.
Step 4: define success
Success criteria should be specific, measurable, achievable and relevant, and usually multidimensional: task fidelity, consistency, tone, privacy, latency and price (Anthropic, define success). Synthesis: separate three layers of metric and agree all three with the customer up front.
| Layer | Examples | Measured by |
|---|---|---|
| Model / task quality | Accuracy vs expert labels, groundedness, refusal rate | Offline evals, LLM-as-judge |
| Operational | p95 latency, cost per task, escalation rate, uptime | Observability |
| Business outcome | Handle time, conversion, hours saved, error cost avoided | A/B tests, before/after baselines |
The sources give examples of business-outcome metrics. Morgan Stanley gated every use case on evals against expert advisors. Indeed measured applications started (+20%) and downstream success, meaning employers were more likely to hire (+13%), against its previous matching engine. Klarna measured resolution time (11 down to 2 minutes) at parity on satisfaction (OpenAI, Enterprise). Capture the pre-AI baseline during discovery. Without it no business metric can be proven later.
Step 5: build vs buy
| Option | When | Source |
|---|---|---|
| Buy a vertical product / partner | Generic need (documentation chatbot, basic text-to-SQL); speed matters | Applied LLMs; vendor partnerships succeed about 67% of the time vs about a third as often for internal builds (MIT NANDA) |
| Build on hosted model APIs | Proprietary workflow or data is the differentiator | Start with inference APIs and self-host later for privacy, control or cost (Applied LLMs) |
| Fine-tune | Prompting and RAG proven insufficient; cost at scale | Indeed’s fine-tuned smaller model used 60% fewer tokens (OpenAI, Enterprise) |
| Train from scratch | Almost never | “No GPUs before PMF” (Applied LLMs) |
Caveat: the MIT NANDA buy-vs-build figure comes from a small preliminary study (its methods note lists interviews with 52 organisations and 153 leader survey responses). Treat it as directional.
Failure modes in discovery
- Solution-first scoping: “we need an agent” before naming the workflow and metric (Gartner’s “agent washing” is the vendor-side mirror) (Gartner).
- Choosing the impressive case over the valuable one. OpenAI warns that complex use cases often slow organisations down (OpenAI, use cases).
- Visible over valuable. Budgets flow to front-office demos while the ROI is in the back office (MIT NANDA).
- No domain expert in the loop. Estée Lauder pairs a business user, a subject-matter expert and a tech lead on every use case (OpenAI, use cases).
- Unmeasurable success: no baseline, no eval set, no agreed threshold.
Deliverable checklist
Synthesis: a discovery phase should end with a one-to-two-page brief (Estée Lauder uses a two-page brief) containing:
- The workflow, its users, its volume and the current baseline.
- The task the LLM performs and what stays human.
- Data sources, access path and sensitivity.
- Success metrics at all three layers, with go/no-go thresholds.
- An initial eval set. Anthropic suggests starting agent evals with 20-50 tasks drawn from real cases (Anthropic, demystifying evals; see verification and evals).
- Risks, guardrails and the escalation path.
- Build/buy decision, time-box and an owner.
Sources
- Identifying and Scaling AI Use Cases (OpenAI)
- A Practical Guide to Building Agents (OpenAI)
- AI in the Enterprise (OpenAI)
- What We Learned from a Year of Building with LLMs
- The GenAI Divide (MIT NANDA)
- Define success criteria and build evaluations (Anthropic)
- Building effective agents (Anthropic)
- Gartner: 40%+ of agentic AI projects canceled by 2027
- Forward Deployed Engineers (The Pragmatic Engineer)
- Demystifying evals for AI agents (Anthropic)