12 questions to ask an AI development company before you sign, and the answers that should worry you
Search for how to choose an AI development company and you'll find plenty of checklists. Nearly all of them are written by agencies, and most stop at "check their portfolio" and "ask about their experience".
That isn't enough for AI work, because an AI system can look finished long before it is. A demo built in a week can answer ten prepared questions perfectly. The hard part is everything after that: wrong answers on real data, running costs nobody predicted, a model provider switching off the model the whole system depends on.
We're an AI engineering company, so read this knowing we're one of the firms you might be comparing. We've tried to write the questions we'd want a client to ask us, including the ones that make agencies uncomfortable. For each question there's what a good answer sounds like, and the answers that should worry you.
Before the questions: what the evidence says
Two things are worth knowing before you speak to any vendor.
Many AI projects get cancelled. In June 2025, Gartner predicted that over 40% of agentic AI projects will be cancelled by the end of 2027, citing cost, unclear business value and weak risk controls. It's a prediction, based on a January 2025 poll of 3,412 webinar attendees, so treat it as a warning rather than a measurement.
The main problem is quality, not cost. In LangChain's State of Agent Engineering survey of 1,340 practitioners (June 2026), quality was the most-cited barrier to production, at about a third of respondents, ahead of cost. Only 52.4% ran offline evaluations, even though 89% had observability in place.
So the questions that matter most are about how a vendor will make the system correct, keep it correct, and keep its cost predictable. Not about how impressive the demo looks.
1. "Can I use something you've built that's live right now?"
Not a case-study PDF or a screen recording. Something live, with real users, that you can try.
A good answer: a URL, a live product or a client reference you can call, plus an honest account of what went wrong after launch and how they fixed it.
Worry if: everything is under NDA, every example is a prototype, or nobody can tell you about a production problem. Every team that has run AI in production has stories. A vendor without any probably hasn't.
Our answer, for transparency: ConGreeto, a multi-tenant AI website assistant we built and run ourselves. There's a case study, and the posts linked through this article cover the production problems we've hit.
2. "Should we build this at all?"
The most useful thing a vendor can say early on is "you don't need us for this part".
A good answer: they ask what you're trying to achieve, then tell you honestly if an existing product covers most of it. For a standard website chatbot or helpdesk bot, a subscription product is usually cheaper and faster than a custom build. Custom work pays off when your data, integrations or rules are unusual, or when the AI is the product.
Worry if: every problem has the same answer, a custom build, and the call skips your business problem and jumps straight to technology.
We set out the build-vs-buy maths in detail in what a RAG chatbot actually costs.
3. "How will we know it's working?"
This is the most important question on the list, and the one most often answered badly.
A good answer: before they write any code, they'll build an evaluation set: a list of real questions or tasks from your business, with the correct answers, which the system is tested against before launch and after every change. They'll tell you which business metric it should move (enquiries captured, tickets resolved, hours saved), not just "accuracy".
Worry if: they quote a benchmark score. Public benchmarks say little about your use case, and some can be gamed. In April 2026, UC Berkeley researchers showed an agent scoring near-perfect on eight major agent benchmarks without solving the tasks. One benchmark scored 100% on an empty response.
Also worry if they rely on "another AI model will grade the answers" with no human checking. A June 2026 study of about 541,000 judgments from 21 judge models (arXiv:2606.19544) found large gaps between how often AI judges appear to agree and how often they agree once chance is accounted for. Model grading is useful, but it needs human spot checks.
4. "What will this cost to run each month, and what's the worst case?"
Build cost is in the quote. Running cost usually isn't, and it's the one that surprises people.
A good answer: a monthly estimate broken down into model tokens, hosting, background jobs and maintenance, with a worst case as well as an average. They'll explain what makes the cost go up (longer documents, more tool calls, bigger models) and what limits it (caching, quotas, smaller models for simple steps).
Worry if: you only get a per-token price, or a single monthly figure with no range.
Why it matters: an April 2026 study (arXiv:2604.22750) found that the same agentic task could use up to 30 times more tokens on one run than another, and that models systematically underestimate their own usage. Large companies get caught out too. Uber reportedly spent its full-year 2026 budget for AI coding tools by April, then brought costs under control with caching, model defaults and dashboards.
We break this down in what an AI agent actually costs to run.
5. "What happens when the model is retired?"
Almost nobody asks this, and in 2026 it's no longer hypothetical.
On 14 October 2026, OpenAI notified developers that GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini will be removed from its API on 14 January 2027. Fine-tuned versions of those models follow the same schedule. Many AI systems built in the last two years run on one of them.
A model change isn't a find-and-replace. When we tested the recommended replacement on our own product's real prompts (two assistants, 22 test cases, October 2026), the cheapest setting broke an instruction that the old model had followed: it replied in a language the assistant had been told not to use. A slightly higher setting fixed it. A small test, but enough to show that a "drop-in replacement" still needs testing against your own evaluation set. See question 3.
A good answer: they tell you which model and version they'll use, how they'll find out about deprecations, who pays for the migration work, and how they'll re-test before switching. Ideally it's written into the support terms.
Worry if: "we'll just update it". Or worse, the contract ends at launch and the migration becomes a new project at full price.
6. "Why this model provider, and can we switch?"
A good answer: a reason tied to your needs (cost, quality on your language or domain, data location, latency), plus a design where the provider can be changed without rewriting the whole system.
Worry if: the provider is chosen because it's the vendor's favourite, or the system is so tied to one provider's features that switching means starting again. Prices, models and terms change several times a year.
7. "What can it do without a person approving it?"
This matters most for agents, the AI systems that take actions (send emails, update records, issue refunds) rather than only answering questions.
The OWASP Top 10 for LLM Applications 2026 (published August 2026) moved Excessive Agency, an AI system with more permissions or autonomy than it needs, from sixth to third place. It's the first edition weighted by real reported incidents.
A good answer: a list of every action the system can take, which ones need human approval, what permissions it has in each connected system (read-only wherever possible), and how actions are logged and reversed.
Worry if: the agent gets an admin API key "to keep things simple", or nobody can tell you what it's allowed to do.
8. "Where does our data go, and who can see it?"
A good answer: a plain description of where your data is stored, which third parties process it (the model provider, the hosting company), whether it's used for training (for most business API plans it isn't by default, but check), and how your data is kept apart from other clients'. If you operate in a regulated market they should know the rules, such as GDPR, Saudi Arabia's PDPL (fully enforceable since 14 September 2024) or the UAE's Federal Decree-Law 45 of 2021, or say so if they don't.
Worry if: "it's all secure" with no detail, or customer data is pasted into tools nobody has approved.
If the system serves many customers from one platform, ask how each customer's data is isolated. We explain why that's a data-model decision, not a filter added later, in multi-tenant AI SaaS architecture.
9. "How does it stay up to date when our information changes?"
An AI assistant that answers from your website or documents is only as current as its last update.
A good answer: a clear process for re-reading changed content, removing outdated content, and deciding which source wins when two disagree.
Worry if: "you just upload the documents". We learned this one the hard way. In our own product, an old uploaded file kept supplying out-of-date prices after the website had been updated, because uploaded documents were trusted over the website. The fix wasn't technical. It was a decision about which source is authoritative, and a way for the owner to see and remove stale content.
More on this in building a chatbot knowledge base from a website.
10. "Who exactly will build it, and who's on call after launch?"
A good answer: the names and roles of the people who'll do the work, whether they're employees or subcontractors, and what happens when something breaks on a Saturday. Response times in writing.
Worry if: senior people on the sales call and unnamed juniors on the project, or support described only as "ongoing maintenance".
11. "What do we own at the end?"
A good answer: you own the code, the prompts, the evaluation sets, your data and the cloud accounts it runs in, or there's a clear licence if any of it stays with the vendor. You can hand it to another team.
Worry if: the system only runs on the vendor's own platform, or the prompts and evaluation sets are "proprietary". Those are often the most valuable part of the work.
12. "What's the smallest paid first step?"
A good answer: a short, fixed-scope first phase (a working pilot on your real data, or a technical discovery) with success criteria agreed up front and a clear point where you can stop.
Worry if: the only option is a large fixed-price contract for the whole thing, quoted before anyone has looked at your data. Or the reverse: open-ended hourly work with no milestones.
The short version
Question
Good sign
Red flag
1. Live work?
URL or reference you can call, plus post-launch problems
Only demos, all NDA
2. Build at all?
Tells you when a product fits
Custom build for everything
3. How do we know it works?
Evaluation set from your data, business metric
Benchmark scores, "it's very accurate"
4. Running cost?
Monthly breakdown with worst case
Per-token price only
5. Model retired?
Named model, migration plan, re-testing
"We'll just update it"
6. Can we switch provider?
Reasoned choice, swappable design
Locked to one provider
7. What can it do alone?
Action list, approvals, read-only by default
Admin keys "for simplicity"
8. Where does data go?
Named processors, isolation, regulations
"It's all secure"
9. Stays up to date?
Refresh process, source priority
"Just upload documents"
10. Who builds and supports?
Named people, written response times
Bait-and-switch team
11. What do we own?
Code, prompts, evals, data
Runs only on their platform
12. Smallest first step?
Scoped pilot with exit criteria
All-or-nothing contract
How to use this list
You don't need perfect answers to all twelve. A small, honest team that says "we haven't dealt with that yet, here's how we'd approach it" is often a better choice than a large one with a polished answer for everything.
Pay most attention to questions 3, 4 and 5. A vendor who handles evaluation, running cost and model changes well has usually dealt with real production problems. A vendor who can't is probably still selling demos.
Frequently asked questions
How do I choose an AI development company?
Ask for live work you can use, an evaluation plan built from your own data, a monthly running-cost estimate with a worst case, and a plan for model retirements. Then compare how honestly each vendor answers, especially when the honest answer is that you should buy a product instead.
What should be in an AI development contract?
Ownership of the code, prompts, evaluation sets and data; named team members; post-launch support with response times; who pays for model migrations; data-processing terms; and a first phase with agreed success criteria and an exit point.
How long does it take to build a custom AI agent?
It depends on the data, integrations and accuracy needed. A focused pilot on real data can often be done in weeks; a system that takes actions in several business tools takes longer, mostly because of testing and permissions. Be wary of a firm timeline given before anyone has seen your data.
What happens to my AI system when OpenAI retires a model?
API calls to a removed model stop working on the shutdown date. For GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini that date is 14 January 2027. Someone has to move the system to a newer model and re-test it, because replacement models can behave differently on the same instructions.
Is it better to hire freelancers or an AI agency?
Freelancers can be excellent for a well-defined piece of work you can manage yourself. An agency makes more sense when you need several skills (data, backend, AI, front end) and someone accountable after launch. Either way, the twelve questions apply.
Sources
- Gartner, Over 40% of agentic AI projects will be canceled by end of 2027, press release, 25 June 2025 (poll of 3,412 webinar attendees, January 2025).
- LangChain, State of Agent Engineering, n=1,340, June 2026.
- UC Berkeley RDI, Trustworthy benchmarks, April 2026.
- arXiv:2606.19544, LLM-as-judge reliability study, June 2026.
- arXiv:2604.22750, token usage in agentic tasks, April 2026.
- Forbes, Uber burns its 2026 AI budget in four months on Claude Code, 17 May 2026.
- OpenAI, Deprecations, GPT-4 series notice dated 14 October 2026, checked 17 October 2026.
- OWASP, Top 10 for LLM Applications 2026, 4 August 2026.
- Model-replacement test: Brillnex Systems, two ConGreeto assistants, 22 test cases, 15 October 2026. A small internal test, not a benchmark.
From us
- What a RAG chatbot actually costs: build, buy or have it built
- What an AI agent actually costs to run in production
- Multi-tenant AI SaaS architecture
- AI agent development: if you're comparing vendors, send us your twelve questions and we'll answer them in writing before any call, including whether you need us at all.