AI Use-Case Discovery

How to find, qualify, scope and prioritise LLM use cases in an organisation: where to look, fit tests for LLMs and agents, impact/effort scoring, success metrics and build-vs-buy.

Contents

AI use-case discovery is the front end of an enterprise LLM engagement. You find the workflows where a language model can create measurable value, test whether an LLM (or an agent) is the right tool for each, define success in measurable terms, and rank the candidates by value and effort. The output is a scoped problem with an agreed metric and a small eval set, ready for a proof of concept. It is the first phase of the forward deployed engineer’s work. OpenAI’s FDEs map processes onsite, prototype and prioritise, then validate the scope with evals before they build (Pragmatic Engineer).

Why it matters

Synthesis: many failed AI projects fail at the selection stage rather than because of the model. Gartner forecasts that over 40% of agentic AI projects will be canceled by end of 2027 because of cost, unclear value and weak risk controls, and advises pursuing agents only where value is clear (Gartner 2025). MIT NANDA found that over half of GenAI budgets go to sales and marketing, while the largest returns come from back-office automation. Startups that target one pain point do well (MIT NANDA).

Step 1: where to look

OpenAI’s guide, drawn from 300 implementations and 600+ use cases, points to three kinds of friction (OpenAI, use cases):

  • Repetitive, low-value tasks, such as summarising, report generation and answering the same questions. Ask people for their “anti-to-do list”.
  • Skill bottlenecks, where work waits for another team’s expertise (data analysis, code, design).
  • Navigating ambiguity, where people get stuck starting open-ended work.

It then classifies use cases into six primitives: content creation, research, coding, data analysis, ideation/strategy and automation. The advanced step is workflow mapping: break a department’s multi-step workflow into tasks and map each task to a primitive.

Discovery techniques in the sources include onsite process mapping (Pragmatic Engineer), workshops and hackathons, and expert-led bottom-up discovery. BBVA rolled out ChatGPT Enterprise and its employees built 2,900+ custom GPTs in five months (OpenAI, Enterprise).

Step 2: does it actually need an LLM, or an agent?

Apply fit tests before scoring. Start with the simplest solution. Optimised single LLM calls with retrieval and examples are often enough (Anthropic, effective agents). OpenAI says agents fit workflows that have resisted deterministic automation (OpenAI, agents guide):

SignalExampleImplication
Complex judgement with exceptionsRefund approvalAgent candidate
Rules that are hard to maintainVendor security reviewAgent candidate
Heavy unstructured dataInsurance claim processingLLM / document processing
Clear, stable rules on structured dataThreshold-based alertsDeterministic code; no LLM

Synthesis: add four questions that sink projects later. Data access: can the system legally and technically reach the inputs? Error tolerance: what does a wrong answer cost, and who catches it? See hallucination. Ground truth: can experts agree on a correct output? Eval tasks must be unambiguous enough that two experts reach the same verdict (Anthropic, demystifying evals). Volume x value: is the task frequent or costly enough to matter?

Step 3: prioritise

OpenAI’s customer teams use an impact/effort quadrant (OpenAI, use cases):

Low effortHigh effort
High valueHigh ROI focusScope and prioritize
Low valueSelf-serviceDeprioritize
  • High ROI focus (high value, low effort): start here to build momentum.
  • Scope and prioritize (high value, high effort): often transformational. Indeed’s job-match explanations took months of iteration and lifted applications started by 20%.
  • Self-service: individual assistants.
  • Deprioritize: for example, rebuilding something an existing tool already does.

Re-score each quarter, because model progress turns high-effort ideas into low-effort ones. The guide also stresses leadership sponsorship and starting with quick wins rather than impressive complex cases.

Step 4: define success

Success criteria should be specific, measurable, achievable and relevant, and usually multidimensional: task fidelity, consistency, tone, privacy, latency and price (Anthropic, define success). Synthesis: separate three layers of metric and agree all three with the customer up front.

LayerExamplesMeasured by
Model / task qualityAccuracy vs expert labels, groundedness, refusal rateOffline evals, LLM-as-judge
Operationalp95 latency, cost per task, escalation rate, uptimeObservability
Business outcomeHandle time, conversion, hours saved, error cost avoidedA/B tests, before/after baselines

The sources give examples of business-outcome metrics. Morgan Stanley gated every use case on evals against expert advisors. Indeed measured applications started (+20%) and downstream success, meaning employers were more likely to hire (+13%), against its previous matching engine. Klarna measured resolution time (11 down to 2 minutes) at parity on satisfaction (OpenAI, Enterprise). Capture the pre-AI baseline during discovery. Without it no business metric can be proven later.

Step 5: build vs buy

OptionWhenSource
Buy a vertical product / partnerGeneric need (documentation chatbot, basic text-to-SQL); speed mattersApplied LLMs; vendor partnerships succeed about 67% of the time vs about a third as often for internal builds (MIT NANDA)
Build on hosted model APIsProprietary workflow or data is the differentiatorStart with inference APIs and self-host later for privacy, control or cost (Applied LLMs)
Fine-tunePrompting and RAG proven insufficient; cost at scaleIndeed’s fine-tuned smaller model used 60% fewer tokens (OpenAI, Enterprise)
Train from scratchAlmost never“No GPUs before PMF” (Applied LLMs)

Caveat: the MIT NANDA buy-vs-build figure comes from a small preliminary study (its methods note lists interviews with 52 organisations and 153 leader survey responses). Treat it as directional.

Failure modes in discovery

  • Solution-first scoping: “we need an agent” before naming the workflow and metric (Gartner’s “agent washing” is the vendor-side mirror) (Gartner).
  • Choosing the impressive case over the valuable one. OpenAI warns that complex use cases often slow organisations down (OpenAI, use cases).
  • Visible over valuable. Budgets flow to front-office demos while the ROI is in the back office (MIT NANDA).
  • No domain expert in the loop. Estée Lauder pairs a business user, a subject-matter expert and a tech lead on every use case (OpenAI, use cases).
  • Unmeasurable success: no baseline, no eval set, no agreed threshold.

Deliverable checklist

Synthesis: a discovery phase should end with a one-to-two-page brief (Estée Lauder uses a two-page brief) containing:

  1. The workflow, its users, its volume and the current baseline.
  2. The task the LLM performs and what stays human.
  3. Data sources, access path and sensitivity.
  4. Success metrics at all three layers, with go/no-go thresholds.
  5. An initial eval set. Anthropic suggests starting agent evals with 20-50 tasks drawn from real cases (Anthropic, demystifying evals; see verification and evals).
  6. Risks, guardrails and the escalation path.
  7. Build/buy decision, time-box and an owner.

Sources

Suggest changes →