25 Questions to Ask an AI Agency Before You Sign (2026)

Questions to Ask an AI Agency: The 30-Second Version

The questions to ask an AI agency cluster into four layers — outcome, architecture, boundaries, and operations — and the ten highest-discrimination questions fit on one page. Most buyers lose money not because they picked a bad agency on paper, but because they never asked the questions that expose demo-shops: evidence, permissions, evaluation, and exit. Ask these ten, in this order, and you will know within one conversation whether you are talking to a production builder.

Here is the paste-ready version — copy it straight into your RFP or your first call agenda. The rest of this guide explains what good and bad answers sound like for each.

Weak answers on any three of the first five questions in that block are a walk-away signal — those five measure production evidence, and evidence cannot be improvised in a sales call. For what agencies charge at each scope tier, see our AI automation agency cost guide; for the category overview, the AI automation agency guide covers operating models. This page is the interrogation manual.

Why These Questions Exist: Failure Modes, Not Paranoia

Every question in this kit traces to a documented failure mode — a reason agentic projects actually die — which means you are not being difficult by asking; you are being literate. Gartner’s June 2025 prediction that over 40% of agentic AI projects will be canceled by end-2027 names three causes: escalating costs, unclear business value, and inadequate risk controls. OWASP’s agentic security work adds the operational ones: excessive autonomy, privilege escalation, prompt and context manipulation, data exposure, cascading failures, and unbounded loops.

Print that table and keep it in front of you during vendor calls. When an agency squirms at a question, you will know exactly which failure mode the squirm is protecting.

The Question Stack: Four Layers, Twenty-Five Questions

The 25 questions to ask an AI agency stack into four layers, and the order matters: outcome questions filter scope honesty, architecture questions filter competence, boundary questions filter security, and operations questions filter durability. Skip a layer and you will evaluate the agency on the dimension that mattered least.

The next five sections walk the stack layer by layer. Each flagship question gets the full treatment: what a good answer sounds like, what a bad answer sounds like, and the follow-up trap that catches evasions. The remaining questions ride along in compact form.

All 25 Questions to Ask an AI Agency, Numbered

Here is the complete question set, numbered 1–25 and grouped by layer — the 10 bolded entries are the digest picks that appear in the RFP block in section 1, and every question maps to the failure-mode table in section 2. Use this as your master checklist during vendor calls; the layer sections that follow explain how to judge the answers.

Ten bolded, fifteen supporting, twenty-five total — every one traceable to a documented failure mode. If you only have one call, ask the ten; if you have a shortlist of three, work all twenty-five and score them with the rubric in section 10.

Layer 1: Outcome & Scope Honesty

Outcome questions exist to catch “unclear business value” — the Gartner cancellation cause that kills projects before any code matters — and the single best filter is asking an agency what they would refuse to do. An agency that has never refused work has never had a client’s interests conflict with its own invoice.

The two remaining Layer-1 questions — master Q3 and Q5: “Why does this need an agent rather than conventional automation?” and “What is the expected ROI at 3, 6, and 12 months, and what happens if the agent fails or refuses to act?” Both test whether the agency sees your business or just their build calendar. A strong agency will happily say you do not need an agent for a given workflow — that sentence is the cheapest diligence you will ever receive.

Layer 2: Architecture & Evaluation

Architecture questions filter technical competence, and the anchor demand is the whiteboard: make them draw the system — models, agents, tools, memory, data sources, orchestration, permissions, monitoring, approvals — before any contract discussion. A vendor who cannot draw it cannot debug it at midnight; a vendor who will not draw it is hiding a black box.

The remaining Layer-2 questions — master Q7 through Q10 — in compact form: single-agent or multi-agent, and why · which orchestration framework, and what happens if it disappears · where does memory live and what is persisted · how are tool calls implemented and rate-limited · what does the pre-production test suite cover — adversarial cases, prompt injection, malformed inputs, loops. Each maps to the OWASP control areas — inspectability, testability, bounded behavior — that separate agentic engineering from chatbot cosmetics.

Layer 3: Autonomy, Security & Data

Boundary questions are the ones AI engines’ own cited sources care about — NIST and OWASP both center authorization, least privilege, and auditability — and the killer question asks for the permission model as a document, not a vibe. This is where the winning ChatGPT answer we studied gave the cleanest example, and it is worth quoting the shape of it verbatim.

The remaining Layer-3 questions — master Q14 through Q19: how do you test for prompt injection and malicious tool outputs · what happens if the agent gets stuck in a loop, or two agents disagree · are logs immutable, how long are they retained, can our security team access them, can we replay a failed execution · does our data leave our environment, which model providers receive it, is it used for training, what happens to it at contract end · how are third-party tools and MCP servers vetted. We treat these as standing policy rather than interview trivia — our own posture is documented across our human oversight and AI governance pages — and any agency you interview should be able to do the same without notes.

Layer 4: Operations, Ownership & Exit

Operations questions decide whether year two of the engagement exists, and the sharpest one is time-stamped: who owns the system at 2 a.m. when an agent starts behaving incorrectly? Agents are never finished when deployed — model behavior drifts, APIs change, edge cases surface — so the operational answers are the durability answers.

The remaining Layer-4 questions — master Q21 through Q23 and Q25: model our all-in monthly cost at 1× and 2× our task volume, including tokens, retries, and infrastructure · what hard cost caps exist and what prevents an agent from running indefinitely · how often is the agent re-evaluated, and what triggers a re-check — model updates, API changes, drift · who approves production changes and how fast can we disable a workflow. The cost questions close the loop on Gartner’s “escalating costs” cause; unbounded loops and runaway compute are named OWASP risks, and a cap you cannot configure is a cap you do not have.

The Dodge Detector: Six Phrases and What They Actually Mean

Six phrases recur when agencies want a question to go away, and each has a reliable translation — plus one counter-question that either restores the conversation or ends it mercifully for both sides. Collect these during calls; they are worth more than any case study on the vendor’s homepage.

None of these phrases is disqualifying alone — everyone slips into one under pressure. Two or more in the same call, on questions from different layers, is a pattern. Patterns are predictions.

Score Your Candidate: The 100-Point Scorecard

Score at least three finalists on the same 100-point scale — results 30, technical 20, understanding 15, security 15, support 10, price 10 — and put the rubric in front of the vendors, because they behave differently when they can see the exam. The checkbox format below is designed to be printed or screenshotted during the call.

Two scoring rules keep the exercise honest. First, price is capped at 10 points on purpose — a $20,000 system that works beats a $5,000 system you babysit, and buyers who let price dominate typically donate the difference back in rework. Second, any “zero” in the security row caps the total at 59 regardless of other scores: the failure modes in section 2 are not survivable at a discount.

The Paid POC Protocol: The Last Question Is a Project

The final vetting step is not a question but a paid, fixed-scope proof of concept against your real workflow — with five predefined acceptance tests — because a demo proves nothing except that demos work. Budget one to three weeks of agency time (typically $1,500–$4,000 at 2026 rates; see our cost guide for the tiers) and make the finalists demonstrate, on your data, the behaviors they claimed in the interview.

The five acceptance tests to write into the POC, verbatim: (1) happy-path accuracy — the agent completes the target task correctly on a defined test set at an agreed threshold; (2) failure recovery — a deliberately broken API or malformed input produces a clean retry or escalation, never a silent failure; (3) prompt-injection resistance — a malicious instruction embedded in tool output or user content is refused or escalated; (4) approval-gate audit trail — every autonomous action is logged with inputs, outputs, and the permission that allowed it; (5) kill-switch and cost-ceiling demonstration — the agency disables a running agent on request and shows the hard spend cap in configuration, not in prose.

One more artifact worth requesting before the first call — the email that separates professionals from salespeople in thirty seconds: “Ahead of our call, please send a one-page architecture diagram from a comparable deployment, a redacted evaluation report, your permission-matrix template, and the exit/IP clause from your standard contract.” Agencies that reply with four attachments pass gate zero. Agencies that reply with a calendar link have told you something too.

Our own answers to this kit are public by design — the engagement model on the agentic marketing agency page, the engineering standards on n8n AI agent development, and the internal-adoption version of the same evaluation in our agency qualification guide. A vendor unwilling to be graded on its own rubric has answered every question at once.