How Do You Evaluate a Process Automation Services Vendor?

I've been on both sides of this conversation. I've sat in the sales call as the buyer, listening to a vendor describe what their platform can do. And I've built the actual systems afterward, including for companies who came to me after a first vendor's promise didn't survive contact with their real documents.

Here's the pattern I've seen enough times to trust it: the pitch is almost always about the platform, the methodology, the years of experience. It is almost never about your actual documents, because most vendors don't want to look at them until you've signed something. That's backwards, and it's the single biggest thing to change about how you evaluate anyone in this category.

Here's what matters most

• Most process automation vendors sell a platform or a methodology. Very few will show you a result on your own documents before you commit to anything.
• Ask for that first. A vendor who can run your actual documents through their system and hand you a written accuracy report, before any contract, is telling you something a sales deck can't.
• Ask specifically whether they're proposing RPA (fixed-sequence automation) or an AI-agent build (reads and adapts to variation). Most won't volunteer this distinction, and it decides whether the system survives your real-world document variety.
• Get the failure mode in writing. What happens when the system isn't sure about a value? If the answer isn't "it stops and asks a person," be skeptical.
• Watch how pricing is explained. "Every engagement is scoped individually" is fine. A number with no explanation of what decides it is a red flag.

Ask to see it work on your documents first

Every vendor in this space has a demo. The demo runs on their sample data, formatted exactly the way their system expects it, because that's the version that always works. It tells you almost nothing about what happens with your documents, which were never built to look like anyone's demo.

What actually tells you something is asking the vendor to run a batch of your own real documents, the messy ones, the inconsistent scans, the vendor invoices that all look different, and show you a written report of what came back. Field by field, with a confidence level on each one. This is the single most useful thing you can ask for in this entire evaluation, because it replaces a sales pitch with a fact. Either the system reads your documents well or it doesn't, and you'll know before you've spent anything.

I built this into how I structure every engagement, precisely because I got tired of watching companies buy a platform on a demo and find out three months later it couldn't handle their actual invoice formats. We run a sample first, on real documents, and hand back a written accuracy report before anyone commits to a build. If a vendor won't do this, or wants payment before showing you anything on your own data, that's a real signal, not a formality to skip past.

Find out if they're actually proposing RPA or an AI-agent build

This is the question most sales calls never get to, because most vendors only sell one of the two and would rather not draw the distinction. RPA automates a fixed sequence against a specific screen layout. It's fast to build and cheap to run, and it breaks the moment a vendor changes their invoice template or a document arrives in a slightly different format. An AI-agent build reads the actual content of a document rather than a fixed position, so it survives exactly the kind of variation that breaks RPA.

Neither one is universally right. If every document you process genuinely looks identical every time, RPA is the cheaper, simpler answer, and paying for more than that is wasted money. If your documents come from many different sources in varying formats, which is true for most companies once you actually count how many different senders and templates are involved, RPA will fail constantly and an AI-agent build is the one that survives. Ask the vendor directly which one they're proposing, and ask them to explain why given what your documents actually look like. A vendor who can't answer that clearly hasn't actually looked at your problem yet.

Get the failure mode in writing

Every automated system gets something wrong eventually. What separates a trustworthy one from a dangerous one is what happens next. Ask directly: when the system isn't confident about a value, what does it do?

The answer you want is that it stops and routes that item to a person for review, rather than guessing and moving on. That review queue is not a limitation, it's the thing that keeps the output trustworthy at real volume. A system that always returns a confident-looking answer, with no visibility into which values it was actually unsure about, is worse than doing the work by hand, because you lose the ability to know when to double-check something. I built this in as a hard requirement on every system I've shipped, and it's the first thing I'd ask about if I were sitting on the buyer's side of the table again.

What real proof looks like, from work I've actually shipped

I don't quote numbers I can't back up, so here's what I can point to directly. A firm producing 1,200+ reports a year cut the time per report by 85% once the system started reading their messy source documents instead of requiring a fixed template. A fintech's accounts-payable team cut manual invoice entry 80%, on invoices that arrive from different vendors in different formats, which is exactly the case that breaks a fixed-sequence tool. A private-equity due-diligence team got through document review 70% faster on data-room files that never look the same twice.

Three different industries, three different document types, the same underlying approach: read the actual document, extract what matters, flag what's uncertain, and never guess. That's the standard I'd hold any vendor to, including myself.

The short version, if you're evaluating someone this week

Ask for a result on your own documents before you sign anything. Ask whether they're proposing RPA or an AI-agent build, and make them justify it against what your documents actually look like. Get a straight answer on what happens when the system isn't sure. And treat vague pricing with the same skepticism you'd treat a vague answer to any of the first three questions, because a vendor who can't explain their price usually can't explain their method either.

Frequently asked questions

What's the single most important question to ask a process automation vendor?

Whether they can run a sample of your actual documents and show you a written accuracy report before you commit to anything. A demo on their own sample data tells you almost nothing about how the system will handle your real, messy documents.

How do I know if I need RPA or an AI-agent build?

Look at whether your documents genuinely look identical every time, or vary across sources and formats. Fixed, unchanging processes fit RPA. Documents that arrive from multiple senders in different formats need an AI-agent build, because RPA breaks on exactly that kind of variation.

What should happen when the system isn't sure about a value?

It should stop and send that item to a person for review rather than guessing. If a vendor can't clearly explain this failure mode, that's worth pressing on before you commit.

Is a lower price always a red flag?

Not by itself, but an unexplained price is worth questioning. Ask what specifically decides the number, document volume, format variety, number of destination systems, and be wary of anyone who can't walk you through it.

How long should a real evaluation take?

Getting a sample accuracy report on your own documents typically takes days, not months. If a vendor's evaluation process takes longer than that just to show you a first result, ask why.

See what this looks like on your own documents

If you want to run this evaluation yourself, send us ten of your real documents and we'll show you what comes back, field by field, before anything is scoped: get an accuracy report. Or see the full approach to how these systems get built.