Pick the agency that can name the workflow it will change and the number that should move, with a date your team first uses it. Tool lists and slide decks tell you very little. Below are the questions I would ask if I were on your side of the table, and how to read the answers.
What should an AI automation agency actually deliver?
A working system inside the tools you already pay for, plus the documentation and training your team needs to run it without calling the agency every week. The first proposal should name one workflow, today's baseline for it and when the new version goes live. If it can't, the agency hasn't done enough discovery to quote.
"Workflow" sounds abstract, so make it concrete. "Inbound leads get researched and routed to the right rep within ten minutes" is a workflow. "AI for sales" is a category. You want the first kind written into the scope.
Which questions show whether an agency can really build?
Ask about failure, because builders have thought about it and resellers usually haven't. What happens when the model is unsure? Which actions wait for a person? Where do the logs live, and who gets the alert when a run breaks at 2 a.m.? Specific, slightly boring answers are a good sign. Polished general ones are not.
Score each answer 0 for no answer, 1 for a generic one and 2 for one that refers to your process. Anything under 8 out of 12 deserves a second conversation before you sign.
| Question | A strong answer mentions |
|---|---|
| What happens when the AI is unsure? | A confidence check, and the person who receives the exception |
| Which actions need a person to approve them? | Specific actions, the approver and any sending or spending limits |
| Where will it run? | Your accounts and API keys, in an environment you can log into |
| What happens when a run fails? | The alert channel, the owner and a written runbook |
| What do we get at handoff? | Code, prompts, documentation, run history and a training session |
| How will we know it worked? | One metric, its current baseline and the date it gets measured |
What does a sensible first project look like?
One workflow, built end to end on real cases. That means discovery with the people who do the work today, a build that uses your actual data, testing on the messy inputs, and a handoff your team signs off on. Ask for dated milestones instead of one delivery date. If the first proposal is a platform rebuild, the scope is too wide.
Who should own the system when the engagement ends?
You should, and the contract should say so before anyone writes code. Spell out the code, prompts, data, accounts, API keys and software licenses, plus who runs the system afterward. Systems built inside an agency's own accounts are the hardest to take back, so ask on day one where everything will live and who can log in.
What are the red flags in a proposal?
A fixed price before anyone has mapped your process is the clearest one. Close behind it: a tool list with no named workflow, results promised without a baseline, and a system that would only run in the agency's accounts. Each of these quietly moves the risk of the project from the agency onto you.
- Price before discovery. Without a mapped process, the number is a guess.
- Guaranteed results. Ask what the baseline is and who measures it.
- No plan for failure. Every automated system breaks sometimes. What matters is who finds out first.
- Agency-only accounts. You should be able to see, and keep, everything that runs.
When is hiring an agency the wrong move?
If a mature product already does the job well enough, buy the product. If the process changes every week, settle the process first. And if nobody inside the company will own the system after launch, name that owner before you hire a builder. An agency can't solve any of these for you, and a good one will tell you so.
How do you run a two-week paid pilot?
Choose one narrow workflow with a clear owner and a number you already track. On day one, agree what success looks like, what data the agency can touch and which actions need approval. At the end you should have a working version running on real cases and a written plan for the full build, whichever agency you choose.
Keep the pilot small on purpose. Its job is to show you how the agency works when something goes wrong, which a sales call never will. See the glossary entry on paid pilots for the short version.
Where does OsmoSync fit?
OsmoSync is an AI consulting and implementation partner. We fit when you need several connected systems built inside your current tools, with a person approving the actions that matter. We don't fit a single off-the-shelf tool setup, or a programme that needs a large on-site team. The systems we run on ourselves are on the showcase.
Further reading
Other perspectives on the same question. Most are written by companies with something to sell, ourselves included, and each one is labelled.
- CIO: What to look for in an AI implementation partner. Trade press.
- Pythian: Choosing the right AI implementation partner. Written by a firm that sells this service.
- AutomateNexus: A 15-point evaluation framework. Written by a firm that sells this service.