Short answer: assess an AI and cloud partner on artefacts, not assurances. Six of them — an evaluation set with a score history, a cost-per-operation figure, a guardrail design, an infrastructure-as-code repository, an observability trace, and a handover package. A partner running real systems produces all six within a couple of days.
Why the two halves must be assessed together
Modern AI projects fail in infrastructure ways at least as often as in model ways: a queue that backs up, a secret that leaks, a spend curve nobody watched, a deployment nobody can reproduce.
Buying the model work from one supplier and the platform from another puts your reliability at the seam between two contracts. Assess both halves in one conversation, and prefer a partner who owns both.
The six criteria and their artefacts
### 1. Evaluation discipline
Artefact: a scored evaluation set with a history.
Fifty to five hundred representative inputs, expected outputs or graded rubrics, run automatically on every change, with a score you can watch over time. Without it nobody can say whether last week's prompt edit helped. This single question sorts the market faster than any other.
Ask: what is the score now, what was it a month ago, and what changed?
### 2. Cost per operation
Artefact: a number in cents, with assumptions.
Per ticket, per document, per decision. Assumptions should include average context size, retry rate, cache hit rate and model tier. "It depends on usage" means they have never paid an inference bill at scale.
Ask also: where are the caps, and are they enforced server-side? A per-user cap enforced in the client is not a cap.
### 3. Guardrails and failure behaviour
Artefact: a written description of the validation and fallback path.
Schema-constrained output with bounded retries. A deterministic or human fallback when the provider is slow or down. Designed refusal behaviour, because "confidently wrong" is the failure mode that costs reputation.
Ask: what does a user see during a provider outage?
### 4. Infrastructure as code
Artefact: a Terraform or Pulumi repository from a previous client, names removed.
The whole environment described in code in your repository, state backend you control, pipeline that deploys on merge, staging that differs from production only in scale and data, and a rollback that has been used deliberately at least once.
Ask: who holds root in our cloud accounts? Only one answer is acceptable — you, with scoped roles for them.
### 5. Observability
Artefact: a production trace, customer data blurred.
Traces and spans, token counts, latency percentiles, error classes, and the ability to replay one user's failed interaction. Probabilistic systems cannot be debugged from logs alone.
Ask: what alerts fired last month, and what did someone do about each?
An alert nobody acts on is a subscription, not a safeguard.
### 6. Exit and portability
Artefact: a handover package from a completed engagement.
Prompts, schemas, tool definitions, evaluation sets and traces in your repository, not in a vendor console. Model behind an interface so swapping providers is a configuration change plus a re-run of evals. Runbooks written for someone who has never met the author.
Ask: how long would it take you to move us from one model provider to another?
Data residency, decided in week one
Where inference runs, what the provider retains, and whether training on your data is contractually excluded. These change the architecture — EU-only inference constrains model choice and latency budgets; zero retention changes how you debug — so they cannot be retrofitted cheaply.
An EU-based partner treats this as a default. For a North American business with European customers, that removes a compliance workstream rather than adding one, and the same architecture satisfies most US state privacy law.
Engagement shape
| Phase | Shape | Why |
|---|---|---|
| Integration build | Fixed scope against an evaluation target | Evals define "done", so a price can hold |
| Model usage | Passed through at cost | You should see the real number |
| Operations | Retained senior team, monthly | You are buying availability |
Ours, published: €9,500 for a bounded fixed-scope slice, €7,000 a month for a dedicated senior team, €450 a senior engineer day, model usage at cost with cost per operation handed to you before launch. Pricing.
The assessment in one meeting
Ask for the six artefacts. Give them two days. What comes back — and how quickly — is the assessment.
We run AI dispatch, concierge and support systems in production, plus surgerank.ai and magicalcalls.com, which you can open right now. See the Agentic AI work or talk to an engineer.
Frequently asked
What single artefact best predicts AI reliability?
A scored evaluation set with a history. Teams that evaluate know whether a change helped; teams that do not learn from users. No other artefact separates the field as quickly.
How should cloud deployment be assessed alongside AI?
Infrastructure as code in your repository, your cloud accounts, a pipeline that deploys on merge, alerts tied to user-visible symptoms, runbooks and a cost dashboard. AI systems fail in infrastructure ways as often as in model ways.
How do I keep AI costs predictable?
Demand a cost-per-operation figure with assumptions before launch, per-tenant or per-user caps enforced server-side, caching where inputs repeat, and alerting on spend anomalies. Inference cost scales with retries and context, not with users.
What exit terms matter for AI work?
Prompts, schemas, evaluation sets and traces in your repository; the model behind a swappable interface; and no dependence on a vendor console you cannot export. If leaving means losing your evals, you do not own the system.