How to assess an AI integration and cloud deployment partner

Short answer: judge an AI integration partner on their evaluation harness, their cost-per-operation figure, and what their system does when the model is wrong. Anyone can produce a convincing demo. Very few can tell you the accuracy, the price per thousand operations, and the fallback path.

The demo is not the product

An LLM demo is the cheapest impressive thing in software. Thirty lines and an API key produce something that looks like the future in a meeting room.

The product is everything the demo omits: what happens when the model returns malformed output, when the provider has an outage, when a user pastes 40,000 words, when the answer is confidently wrong, when spend triples because a loop retried, and when someone asks six months later why a particular decision was made.

So the assessment criteria below are all about the omitted parts.

1. Do they evaluate, or do they vibe?

Ask to see an evaluation set. A real one: fifty to five hundred representative inputs with expected outputs or graded rubrics, scored automatically, run on every change.

Without it, nobody can tell whether last week's prompt edit improved things. Teams without evals do not know their system got worse; they find out from users. This single question separates the field faster than any other.

Follow-up worth asking: what is your current score, and what was it a month ago?

2. Cost per operation, before launch

Every AI feature has a unit: per support ticket, per document, per call, per dispatch decision. A partner who has run a system in production will give you that number in cents, with the assumptions — average token count, retry rate, cache hit rate, model tier.

If the answer is "it depends on usage", the system has not been operated. Costs in AI features do not scale linearly with users; they scale with retries, context size and the long tail of pathological inputs, and all three are knowable in advance.

Budget the pathological cases. The median request is never what breaks the invoice.

3. Guardrails and fallbacks

Three specific questions:

• Schema-constrained output. Is the model's response validated against a schema, with a bounded retry, before it touches your system? Free-text parsing in production is a decision to accept random failures.
• Fallback path. When the provider is down or slow, what does the user see? A graceful degradation to a human queue or a deterministic path is the mark of a system designed for a Tuesday outage.
• Refusal behaviour. What does it do when it does not know? "Confidently wrong" is the failure mode that costs reputation, and it is a design choice, not a model property.

4. Data handling and residency

Decide before architecture: which data may leave your boundary, whether inference must run in the EU, how long prompts and completions are retained, and whether provider training on your data is contractually excluded.

These are not compliance paperwork — they change the design. EU-only inference constrains provider and model choice. Zero-retention changes how you debug. Retrofitting either is a rebuild, so it belongs in week one.

A European partner handles GDPR as a default rather than a project. For North American businesses that is a genuine advantage rather than a complication: you get EU-native data handling for your European users without a separate workstream.

5. Portability

The model you pick today will not be the best model in nine months. That is not a risk to manage, it is a certainty to design for.

The test: is the model behind an interface? Are prompts, schemas and tool definitions in version control rather than in a vendor console? Can the evaluation suite run against a different provider without modification? If yes, swapping is a config change and an afternoon of evals. If no, you have bought a dependency wearing a feature costume.

6. Who owns the operational surface

Traces, spans, token counts, latency percentiles, error classes, and a way to replay a single user's failed interaction. AI systems fail probabilistically, and you cannot debug a probabilistic system from logs alone.

Ask what their observability stack looks like and to see a real trace, with the customer data blurred.

7. How the engagement is shaped

Our own position, for transparency: we run AI integration as a fixed-scope bounded slice at €9,500, with model usage passed through at cost and the cost-per-operation figure given to you before launch. Continuing work is €7,000 a month for a dedicated senior team; individual changes are €450 a senior day. The numbers are published.

Fixed scope works here specifically because the evaluation set defines "done". Without one, "the AI works well enough" is an opinion, and opinions cannot close a contract.

The short checklist

1. Show me your evaluation set and its current score.

2. What is the cost per operation?

3. What happens when the model returns garbage?

4. What happens when the provider is down?

5. Where does inference run, and what is retained?

6. How long would it take you to swap model providers?

7. Show me a production trace.

Seven questions, one meeting. A partner running real systems answers all seven without preparation. Talk to us about a build, or read what we have running in production.