What I Require From Vendors Before an Enterprise AI Pilot

By Priya Sharma, Enterprise AI & Governance Editor

Pilots fail quietly. The slide deck looks green, the champion changes jobs, and the contract auto-renews into a zombie integration nobody wants to unwind. I have sat on both sides of those tables. Before I bless a pilot, I require answers that fit on one page—not a mutual NDA theater marathon.

Non-negotiables I put in writing

Data boundaries

• What customer data leaves our boundary, in what form, and for how long?
• Is training on our prompts or outputs off by default? How is that attested?
• Who can access logs, and is access logged?

If the vendor cannot describe the data path without a marketing metaphor, I do not start.

Evaluation ownership

I refuse pilots whose only success metric is “users liked it.” I want:

• A task list with 20–50 real examples from our domain.
• A baseline (current process time/error rate).
• A decision date and the person who can kill the pilot.

Vendors may help build the eval. They do not grade their own report card alone.

Human accountability

Who is the named internal owner? Who is the vendor technical contact with escalation hours? What happens on a Sev-1 hallucination in a customer-facing flow?

I think “the model said so” is not an accountability model. It is a shrug.

Exit ramp

Before kickoff I want:

• Export format for prompts, configs, and evaluation sets we created.
• Deletion timeline for our data after exit.
• Cost to pause vs. terminate mid-term.

No exit ramp means the pilot is already a marriage.

Questions that expose vapor

• Show me a failure case from another customer (redacted). If they claim zero failures, I assume zero production use.
• Walk me through a refused request and how the product behaves.
• Which features in the demo are GA vs. roadmap? I annotate the demo script accordingly.

How I structure a 6-week pilot

Weeks 1–2: data path review and eval set freeze. Weeks 3–4: limited users, heavy logging. Week 5: compare against baseline with the named decision maker in the room. Week 6: go / no-go with written rationale.

I do not extend “just two more weeks” without a new hypothesis. Endless pilots are how budgets die.

On procurement pressure

Sales urgency is not a risk control. If legal or security needs another week, the calendar moves. I have never regretted a delayed pilot. I have regretted a fast one.

Bottom line

An enterprise AI pilot is a governed experiment, not a vibe check. Require clear data paths, owned evaluations, named humans, and a real exit. Everything else is decoration on a contract.