How to Evaluate an AI Vendor

Every AI vendor sounds the same in the sales call

They all use the same buzzwords. They all show impressive demos. They all claim to have “deep expertise.” They all promise transformation.

The difference shows up in week 8 when the demo breaks on your real data, the timeline has slipped by 3 months, and the quote has doubled.

I’m going to give you the evaluation framework I’d use if I were in your seat. The questions that separate builders from talkers. The red flags that tell you to walk away. And the green flags that tell you you’ve found someone worth hiring.

The 10 questions to ask

1. “Can you show me a production system you’ve shipped?”

Not a demo. Not a slide deck. A running system with real users that’s been in production for at least 3 months.

Good answer: “Yes, here’s the system, here’s what it does, here’s how many users, here’s the uptime.” They can talk about what broke and how they fixed it.

Bad answer: “We can’t share client details due to NDAs.” Or: “Let me show you this demo environment.” If they can’t show you anything real, they haven’t shipped anything real.

2. “What happens when the model is wrong?”

Good answer: “We build fallback paths, confidence thresholds, and human override into every system. Here’s how it works…” They have a specific answer with specific mechanisms.

Bad answer: “We tune the model to be very accurate.” Or: “That’s a rare case.” If they don’t have a fallback plan, they’re building a demo. Production systems have error handling. Demos don’t.

3. “What data do you need from us, and how do you check if it’s good enough?”

Good answer: They ask specific questions about your data sources, formats, completeness, and ownership. They want to see samples. They have a data assessment process.

Bad answer: “We can work with any data.” Or: “Don’t worry, AI is very flexible.” If they don’t ask about data quality, they’ll fail. Data quality is the #1 predictor of project success.

4. “Who owns the code after the project?”

Good answer: “You do. Everything goes into your repository. You have full access from day one.” In writing, not just verbally.

Bad answer: “We retain ownership but you get a license.” Or: “We’ll hand over the code after final payment.” If they hold the code hostage, you’re building a dependency, not a capability.

5. “What does it cost to run after you leave?”

Good answer: A real number. “API costs will be approximately €500/month at your volume. Infrastructure is €200/month. Monitoring is included. Plan for €15-20K/year in ongoing costs.”

Bad answer: “It depends on usage.” Or: “We can discuss a maintenance retainer.” If they can’t give you a number, you’ll find out the hard way.

6. “How is our team trained during the project?”

Good answer: “Your team shadows from day one. They attend standups, review code, and gradually take over operations. By the time we leave, they’ve been running it for weeks.” With a specific phased plan.

Bad answer: “We’ll do a handover session at the end.” Or: “Your team isn’t ready for this yet.” If they don’t have a training plan, they’re not planning to leave.

7. “What’s your approach to EU AI Act compliance?”

Good answer: They know the risk tiers. They classify your use case. They build compliance documentation into the project. They discuss it unprompted.

Bad answer: “Don’t worry about that, it only affects big companies.” Or: “We’ll handle compliance at the end.” If they dismiss compliance, they’re putting you at legal risk.

8. “Can I talk to a previous client whose system is in production?”

Good answer: “Yes, here’s their contact info.” They have clients willing to vouch for them.

Bad answer: “Our clients prefer confidentiality.” Or: “We can arrange a call once we’re further along in the sales process.” If no previous client will talk to you, there’s a reason.

9. “What projects have you turned down?”

Good answer: They can name specific types of projects and explain why. “I turned down a custom model project last month because the data wasn’t ready. Recommended an API approach instead.”

Bad answer: “We haven’t turned any down.” If they’ve never turned down a project, they don’t have standards. They have a quota.

10. “Explain what you’d build for us in one sentence.”

Good answer: Plain language. “We’d connect an AI model to your document database so your team can search your internal knowledge by asking questions in plain English.”

Bad answer: Jargon. “We’d deploy a RAG-based architecture with vector embeddings and semantic retrieval over your knowledge corpus.” If they can’t explain it simply, they don’t understand it well enough - or they’re hiding something.

Red flags

Walk away if you see any of these:

• Starts building before auditing your data. They should want to understand your data before committing to a solution.
• Quotes a price before understanding your data. “€50K for an AI system” before looking at your data is a sales tactic, not an estimate.
• Uses proprietary tools or frameworks. If the tool has their company name on it, you can’t use it without them.
• No handover plan. Or a “handover plan” that’s a meeting, not a process.
• “Your team isn’t ready” without a training plan. This is a stall, not an assessment.
• Can’t show production work. Demos are easy. Production is hard. If they can only show demos, they’ve only built demos.
• Dismisses compliance. “Don’t worry about the EU AI Act” is legally reckless advice.

Green flags

You’ve found a good vendor if:

• They start with questions about your data. Data-first means build-second.
• They tell you what NOT to build. Honesty about what won’t work is more valuable than promises about what will.
• They have turned down projects. Standards mean they care about outcomes, not just revenue.
• They can show running systems. Production proof beats demo promises.
• They have a phased handover model. They’re planning to leave, not to stay forever.
• They discuss compliance unprompted. They understand the legal landscape and protect you from risk.

The vendor matrix

Score each vendor on five dimensions:

1. Technical depth - Can they explain their work simply? Have they shipped production systems?
2. Production track record - Real systems running with real users, not just demos.
3. Transparency - Clear costs, clear timeline, clear handover. No vagueness.
4. Handover commitment - Phased training, documentation, code ownership. In writing.
5. Compliance awareness - EU AI Act knowledge, GDPR considerations, security practices.

Score each 1-5. If any dimension scores below 3, keep looking.

The pilot test

Before signing a big contract, run a small fixed-scope pilot. 2-4 weeks. Fixed price. €5K-€15K.

The pilot should produce something tangible: a data assessment, a working prototype on your real data, or a feasibility analysis. Not a slide deck.

If they deliver, scale the engagement. If they don’t, you’ve spent €10K learning they can’t - not €150K.

What I’d look for if I were you

Someone who’s shipped. Someone who starts with diagnosis, not with building. Someone who plans to leave. Someone who explains things plainly. Someone who has turned down work. Someone who can show you a running system.

If you find that person, hire them. They’re rare.

FAQ

How many vendors should I evaluate? 3-5. More is procrastination. You’re looking for a partner, not running a procurement process. Ask the 10 questions, score the matrix, make a decision.

Should I ask for a free pilot? No. Free pilots are sales calls dressed up as work. Pay for a small fixed-scope engagement. It separates vendors who want to work from vendors who want to sell.