A guide to asking the questions vendors don’t want to answer — before you sign, not after.
By William, U.S. Army (Retired)- Owner, Chief Editor & CEO
A consulting team once ran a pilot for a client evaluating an AI tool for customer support ticket routing. The vendor’s demo was genuinely impressive — the system categorized a dozen sample tickets accurately, in real time, in front of the room. Everyone was sold.
Then they ran it against 500 actual tickets pulled from the client’s live system. Accuracy dropped from 95% to roughly 68%. The real tickets had typos, slang, abbreviations, multi-issue threads, and context the model had no way to know — exactly the kind of mess that never makes it into a vendor’s curated demo.
That gap — between what a vendor shows you and what the system actually does once it’s wired into your real data — is the central problem in AI procurement right now. And it’s not an accident. It’s the business model. Demos are choreographed. Edge cases are pre-screened out. The room is designed to make you say yes.
This article is for anyone evaluating an AI vendor in 2026 — whether you’re running a formal RFP or just trying to figure out if the slick platform your team found is actually as good as it looks. The goal isn’t to make you cynical about AI vendors. It’s to give you the specific questions that separate a vendor who can deliver from one who can only present.
Key Takeaways
- A strong demo and a working production system are not the same thing, and the gap between them is where most enterprise AI budgets get wasted.
- The riskiest AI vendor evaluations use a traditional software RFP, which misses most of the risk categories specific to AI: data routing, model training rights, and explainability.
- Five consistent red flags show up across hundreds of real vendor evaluations: vague data handling answers, no working integration demo, unexplainable model behavior, unpredictable pricing units, and data lock-in.
- The single highest-leverage thing you can do is test the vendor’s system against your own messy, real data before signing — not their clean sample set.
- A credible vendor can describe their implementation methodology, name your actual point of contact post-sale, and produce specific documentation on request. A vendor who can’t do any of those three is telling you something important.
Why Standard Procurement Frameworks Miss the Real Risk
Most companies still evaluate AI vendors the way they evaluate any SaaS tool: build a feature matrix, compare pricing, check a security box, and sign. That approach was built for a world of on-premise software and straightforward SaaS — and it breaks down badly against AI vendors, for three specific reasons.
AI systems process your sensitive business data using probabilistic models you don’t fully control. Many route that data through third-party LLM APIs without making this explicit in the sales conversation. And — increasingly in 2026 — they take autonomous actions at a speed and scale that traditional software simply didn’t. A typical enterprise evaluation using a standard IT RFP misses up to 60% of the risk-relevant questions specific to AI procurement, according to one enterprise AI RFP framework. The result is often a contract signed before security has confirmed where the data actually goes, or a deployment legal later flags for missing compliance documentation.
This isn’t a hypothetical concern padding out a sales page. 72% of S&P 500 companies now flag AI as a material risk in their 10-Ks, up from just 12% in 2023 — a sign that the risk has moved from theoretical to something boards are formally tracking.
The Real Reason Most AI Pilots Never Reach Production
Before getting into specific red flags, it’s worth understanding the structural pattern almost every failed AI procurement shares. It isn’t usually that the model was bad. It’s that the conditions under which it was evaluated had nothing to do with the conditions it would actually operate in.
A typical pilot gets built over four to six weeks in a controlled environment, demoed with curated data and a stable connection, in front of a few friendly colleagues who already know how to use it. The demo goes well. The budget gets approved. Then the team moves toward production, and the production database turns out to have over a dozen different date formats, the ERP system runs an API version the pilot was never tested against, and the interface confuses the actual end users.
The scale of this pattern is larger than most procurement teams assume. A March 2026 survey of 650 enterprise technology leaders found that 78% of organizations have AI agent pilots running — but only 14% have successfully scaled an agent to organization-wide use. That gap isn’t primarily a technology problem. It’s an operating model gap — and a buyer who only ever sees the pilot has no way to know whether the vendor in front of them is one of the 14% or one of the 64% that stalls.
This is exactly why a vendor’s demo tells you almost nothing about what you actually need to know.
Five Red Flags That Show Up
These patterns aren’t theoretical. They come from documented, recurring cases across dozens of AI procurement engagements. The same warning signs surfacing again and again, regardless of industry or use case.
1. Vague or deflected data-handling answers. “Your data is secure and encrypted” is not an answer to “where is our data processed?” If a vendor cannot or will not provide a specific answer about data processing location, that’s a disqualifying red flag for any regulated deployment. The same caution applies to model training: if your data is used to train the vendor’s models by default, with opt-out buried in an obscure setting, assume your data is in the training pool unless you’ve actively confirmed otherwise — including retroactively.
2. No working integration demo with your specific tools. A slide showing a generic CRM logo is not proof of integration. The standard worth holding vendors to: “Show me a live integration with your specific CRM or ERP — not a slide, a working demo.” Vendors who can show this confidently have actually built it. Vendors who can’t are asking you to fund their integration work as part of your implementation timeline.
3. The vendor can’t explain how the model actually works. This is subtler than it sounds, and it’s not about demanding proprietary architecture details. One consultant evaluated a “proprietary AI” document classification tool and asked the vendor how the model was trained, what data it used, and how it handled edge cases. The answers were vague references to “deep learning” and “neural networks,” with no specifics on training data, confidence scores, or failure modes — and it later turned out to be a fine-tuned open-source model with minimal customization. That’s not necessarily disqualifying on its own. The inability to explain it clearly is the actual problem.
4. Pricing units that are impossible to forecast. Vendors who price per API call, per document processed, or per “AI credit” are using units that make cost genuinely difficult to predict at scale. Ask for a worked example using your actual expected volume, not a hypothetical.
5. No real path to get your data back out. If you can’t extract your data from the vendor’s system in a standard format, you are locked in — and lock-in risk in AI procurement isn’t just technical. It includes the organizational cost of having retrained your workforce around a specific vendor’s workflows, which makes switching far more expensive than a simple data migration would suggest.
The Questions That Actually Separate Vendors
Generic procurement questions get generic, rehearsed answers. These don’t.
On data handling: “Walk me through exactly what happens to our data from the moment it enters your system to the moment we get a result.” If the answer is mostly buzzwords, that’s informative on its own.
On production readiness: “Show me a reference customer who is running this in production, at our approximate scale, for at least six months — and let us talk to them without you on the call.”
On failure modes: “How does your system handle a scenario it hasn’t seen before? Show me an example of it failing, and what happened next.” A vendor who has never seen their own system fail either hasn’t tested it seriously or isn’t being straight with you.
On accountability: “Who is the named team that implements this — not the sales team, the actual delivery team — and can we meet them before signing?” The people who sell the implementation and the people who deliver it are frequently not the same team, and that mismatch is a leading cause of post-signature disappointment.
On methodology: “Walk us through your implementation methodology in detail.” A credible vendor can describe a diagnostic phase that precedes any solution design, milestone-based delivery with defined exit criteria, and a defined support model for the period after go-live. Proposals that begin with technology selection rather than process mapping are a consistent early warning sign — it means the vendor is optimizing for their own delivery model, not your actual operational outcome.
The One Test That Matters More Than Any Question
Every red flag above is useful. None of them substitutes for this: before you sign, get the vendor’s system running against a sample of your actual, messy, unfiltered data — not their demo dataset, and not a cleaned-up version your own team prepared to make the pilot look good.
This single step is the clearest predictor of whether a vendor will perform in production. The 95%-to-68% accuracy collapse mentioned earlier didn’t happen because the vendor lied. It happened because nobody tested against real conditions until after the sale. A test against your real data — typos, multi-issue threads, inconsistent formats and all — will tell you more in an afternoon than six weeks of polished demos.
If a vendor resists this test, or insists on supplying or pre-cleaning the sample data themselves, treat that resistance as the answer.
Frequently Asked Questions
What’s the difference between evaluating an AI vendor and evaluating regular software?
Regular software procurement focuses on features, uptime, and price. AI vendor evaluation has to additionally cover where your data is processed, whether it’s used to train models, how the system behaves on data it wasn’t designed for, and what happens when it’s wrong. A standard IT RFP misses most of these risk categories because they didn’t exist before AI vendors became common.
How long should I expect a proper AI vendor evaluation to take?
A thorough process typically runs six to ten weeks: one to two weeks to scope and distribute requirements, two to three weeks for vendor responses, one to two weeks to score and shortlist, and two to three weeks for demos, reference checks, and contract negotiation. Rushing this is one of the most common causes of post-deployment problems, and fixing those problems later typically costs more than the time saved by skipping steps.
What’s the single best way to spot a vendor that will fail in production?
Test their system against your own real, unfiltered data before signing — not a curated sample. The gap between demo performance and production performance is consistently the largest predictor of whether a vendor will deliver, and it’s also the easiest thing to test directly rather than take on faith.
Should I trust a vendor’s reference customers?
Trust references who can speak specifically to post-deployment operational performance, not just the sales process. A reference who says “the demo was great” tells you nothing. A reference who can describe what broke six months in, and how the vendor responded, tells you what you actually need to know.
Is it a red flag if a vendor can’t commit to specific ROI numbers before any diagnostic work?
It’s the opposite of a red flag — it’s a good sign. Vendors who guarantee specific outcomes before they’ve assessed your data, processes, and integration environment are either working from optimistic assumptions or a sales script. A vendor who wants to run a diagnostic before committing to numbers is usually the more credible one.
The Demo Was Never the Point
Every vendor you talk to has built a demo designed to make you say yes. That’s not dishonest — it’s how sales works. The mistake is treating the demo as evidence rather than as a starting point for the questions that actually matter.
The vendors worth signing aren’t the ones with the most polished presentation. They’re the ones who can explain their failure modes without flinching, name the actual team that will implement your project, and let you test their system against the messy reality of your own data before you’ve committed to anything.
That’s a slower process than approving the vendor with the best slide deck. It’s also the difference between a deployment that works in month six and one that’s still “almost ready” a year later.
Keep Learning
- Are Agentic AI Phones Actually Ready to Take Over Your Smartphone?
- Why Do More AI Tools Make Your Team Less Productive?
- Should You Use AI to Save Time at Work, or to Solve Bigger Problems?
- What Is Agentic AI Ransomware, and How Do You Protect Yourself From It?
- How Fast Is Corporate AI Adoption Really Growing Right Now?