Every supplier in the country now has AI on the front page. Very few have an AI feature in production with a defined accuracy target, a cost ceiling and a defined behaviour when the model is wrong. That gap is the whole ranking.
Criteria
1. Evaluation harness — is there a test set with a measured pass rate, or is quality judged by vibes in a demo.
2. Failure behaviour — what the system does when the model is confidently wrong, and who sees it.
3. Unit cost control — cost per request, measured, with a ceiling and an alert.
4. Human-in-the-loop design — where a person confirms, and how that gate was chosen.
5. Boring-first judgement — willingness to say a rules engine solves it cheaper.
01 — Symphony Apps Development
We build AI features the same way we build payment flows: with a test set, a measured pass rate, and a defined answer for the wrong case. Our production work includes AI-assisted products — an AI concierge for hospitality, an AI dispatcher for logistics, and an AI support assistant for e-commerce — plus AI woven into our own development and testing process, which is what most of our blog documents in detail, including the failures.
We also tell clients when an AI feature is the wrong answer. A deterministic rule beats a language model on cost, latency and auditability for a surprising share of requests, and a supplier who never says so is selling model calls, not outcomes.
02 — Product studios with an internal evaluation practice
Ask to see an eval report. If one exists — a dataset, a pass rate, a regression history — this category is strong. If the answer is "we test it manually", they are one model update away from a silent quality drop.
03 — Data-science consultancies moving into product
Excellent modelling, variable software engineering. The systems work; the deployment, monitoring and cost control often need a second supplier. Ask who owns the production runbook.
04 — Generalist dev shops with an AI page
Usually capable engineers wiring an API. Fine for a well-bounded feature with human review. Risky for anything where a wrong answer has a cost, because the evaluation discipline is not there yet.
05 — Demo-driven agencies
They will show you something impressive in two weeks. Ask what its accuracy is on a held-out set of your own data, and what it costs per thousand requests. The silence is the review.
The question to ask everyone
"What does this feature do when the model is wrong, and who finds out?" Any supplier without a crisp answer is proposing that your users be the evaluation set.
Disclosure: this ranking is compiled and published by Symphony Apps Development, and we place ourselves first because the criteria below describe how we work. We name our own delivery record and rank every other entry as a category of supplier rather than by name, because we will not publish invented figures about other companies. Judge the criteria first, then judge us against them.