Situational AI: Research the Need Before You Reach for the Model

1 · The problem: capability-first adoption

The dominant failure mode in applied AI is not technical. RAND's interview study of AI practitioners found the leading root cause of project failure to be misunderstanding — or miscommunicating — what problem needs to be solved, before any question of model quality arises^[1]. MIT's 2025 industry report reached a compatible conclusion from the buyer's side: roughly 95% of enterprise GenAI pilots produced no measurable P&L impact, not because models were weak but because deployments didn't attach to a workflow where they could compound^[2]. Gartner projected that at least 30% of generative AI projects would be abandoned after proof of concept^[3], and S&P Global found the share of companies scrapping most of their AI initiatives jumped from 17% to 42% in a single year^[4].

These numbers describe capability-first adoption: the organization begins with the technology and searches for a place to apply it. The failure literature is remarkably consistent that the inversion — outcome first, mechanism last — is where value comes from. Davenport and Ronanki observed as early as 2018 that companies succeeding with AI took "incremental rather than transformative" steps aimed at specific, well-understood tasks^[5]. Nothing in the generative era has invalidated that finding; the pilot data above suggests it has been forgotten.

2 · Research the need, then choose the mechanism

UX research already owns the tools this problem requires: user interviews, contextual inquiry, journey mapping, service blueprinting. What changes in an AI context is only what we ask the research to decide. Before any build decision, discovery should produce three artifacts:

• A validated outcome statement. Not "add a chatbot" but "new arrivals reach full productivity in 2 days instead of 3 weeks." The outcome must be observable and measurable without reference to any technology.
• A task characterization. Are inputs structured or unstructured? Are the rules enumerable or judgment-laden? How variable is the request distribution? This — not enthusiasm — determines whether the task is automatable, assistable, or delegable.
• An error economics profile. What does one wrong output cost, who absorbs that cost, and how quickly is it detected? Probabilistic systems will err; the question is whether the context can afford it.

Google's People + AI Guidebook encodes the same discipline as a design gate: find the intersection of user need and AI strength, and explicitly ask whether a rules-based solution would suffice before committing to a learned one^[6]. Sculley et al. supply the engineering rationale: machine-learned components carry a distinctive, compounding maintenance debt — entanglement, hidden feedback loops, data dependencies — that deterministic code does not^[7]. An AI you didn't need is not a neutral choice; it is a liability you volunteered for.

3 · Customer-facing vs. operations-facing: different error economics

A second discipline the failure data rewards is distinguishing who absorbs the system's mistakes. Operations-facing deployments (internal triage, drafting, summarization, routing) place errors in front of trained staff who can catch and correct them — the human remains in the loop by default, which is precisely the condition under which imperfect automation performs best^[8]. Customer-facing deployments place errors in front of people with no context, no obligation to forgive, and — in government services especially — real consequences attached to wrong answers. MIT's pilot data shows the market has quietly priced this in: back-office deployments produced the strongest measured returns while high-visibility customer pilots stalled^[2].

4 · AI is not automation — and most problems are automation problems

The industry uses "AI" as a superset that quietly swallows its cheaper neighbors. The distinction matters operationally. Automation executes explicit rules over structured inputs: deterministic, testable, auditable, with marginal costs near zero. AI infers a response from patterns: it handles unstructured and variable input, and pays for that flexibility with probabilistic output, evaluation burden, and the maintenance debt profile described above^[7]. Between them sits an older option the hype cycle skips entirely: a better workflow — eliminating steps rather than accelerating them. Parasuraman, Sheridan and Wickens formalized this as a spectrum of automation levels to be selected per task, not maximized per system^[8]; Bainbridge's classic warning about the "ironies of automation" — that automating the easy parts makes the residual human role harder — applies to agentic systems with fresh force^[9].

5 · A decision sequence

The instruments above compose into a sequence a team can actually run. Its purpose is not to forbid AI but to make each escalation earned by evidence:

6 · Two cases from the field

The automation that looked like an AI problem

A government onboarding process I researched took 2–3 weeks, 15 manual steps, and 3–4 hours of staff time per user. The instinctive 2020s answer is a chatbot. Discovery said otherwise: the dominant causes were unclear role-based requirements and untracked handoffs — rule-shaped problems. Role-based pathways, standardized digital forms, and automated routing (rungs 1–2) cut onboarding to roughly two days with no AI at all. Only after that deterministic floor existed did an agentic layer become worth building — to interpret free-form questions and guide arrivals through the now-reliable pipeline. The agent earned its rung; it did not start there.

The AI that earned its place

Service-desk intake is the opposite characterization: inputs are unstructured plain language, infinitely variable, and judgment-laden at the margins — a poor fit for enumerable rules. There, an AI-operated desk resolving requests from an approved knowledge base, with routing to a human whenever judgment is required, sits deliberately in Fig. 2's guarded quadrants: constrained sources, designed escalation, humans owning the consequential calls. The AI is doing a job automation genuinely cannot — which is the only reason it's there.

7 · Conclusion

The failure statistics in Fig. 1 are not an argument against AI; they are an argument against unconditional AI. The organizations on the right side of those numbers treat AI as one rung on a ladder of mechanisms, entered from the bottom, climbed on evidence, and audited against an outcome that was defined before any vendor demo. That is a research discipline before it is an engineering one — which is precisely why product teams that still invest in discovery are the ones whose AI ships and stays shipped.

References

1. Ryseff, J., De Bruhl, B. F., & Newberry, S. J. (2024). The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed. RAND Corporation.
2. MIT NANDA (2025). The GenAI Divide: State of AI in Business 2025. Massachusetts Institute of Technology.
3. Gartner (2024). Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept by End of 2025. Press release.
4. S&P Global Market Intelligence (2025). AI adoption and implementation survey — share of companies abandoning most AI initiatives rose from 17% (2024) to 42% (2025).
5. Davenport, T. H., & Ronanki, R. (2018). Artificial Intelligence for the Real World. Harvard Business Review, 96(1), 108–116.
6. Google PAIR (2019–). People + AI Guidebook. Google Research.
7. Sculley, D., Holt, G., Golovin, D., et al. (2015). Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems 28 (NeurIPS).
8. Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). A Model for Types and Levels of Human Interaction with Automation. IEEE Transactions on Systems, Man, and Cybernetics — Part A, 30(3), 286–297.
9. Bainbridge, L. (1983). Ironies of Automation. Automatica, 19(6), 775–779.