AI proof of concept: how to frame one so you don't waste your time

An AI proof of concept exists to answer one question: can this be done with our data, and does it deliver enough value to keep going? Nothing more. If by the end you can’t answer that with a yes or a no, the test wasn’t framed well, no matter how well the technology worked.

Most AI tests that fizzle out don’t fail because of the model. They fail because nobody decided up front what was being checked. Someone builds something in an afternoon, a demo comes out that impresses in a meeting, and from there nobody can say whether it was really feasible or just a lucky fluke with three cherry-picked examples. A well-run PoC exists to cut through that ambiguity before it eats your budget.

What an AI proof of concept is (and what it isn’t)

A proof of concept, or PoC, is a small, bounded experiment to check whether an idea is feasible before you commit real time and money. The concept isn’t new or exclusive to AI: in any software project it serves to reduce the biggest technical uncertainty before building anything serious.

In AI that uncertainty is usually a very specific question. Can the model classify our customer emails well? Can it pull the right data from our invoices, which don’t all share the same format? Does it summarize reports without inventing figures? The PoC takes that question, isolates it, and answers it with real data in a short time.

What a PoC isn’t is worth being just as clear about. It isn’t a production rollout dressed up as a test. It isn’t the demo a vendor shows you with their own examples. And it isn’t a toy that a couple of curious people use now and then to “see how it goes”. Those three things get confused with a PoC constantly, and all three lead you to decide on false information.

A PoC is not a pilot

Plenty of people skip this distinction and then end up measuring the wrong thing.

A PoC answers “can this be done?”. It checks technical feasibility: whether the model, with your data, can do the task at acceptable quality. It’s short, few people see it, and it touches no real customer.

A pilot answers a different question: “does this pay off in real work?”. It comes once you already know the technology works and you want to see whether, put to work with real users over a period of time, it improves a business number without breaking something else. The pilot is longer, has real users, and takes on risks the PoC deliberately avoids.

Order matters. A pilot before a PoC means betting money and exposing customers without even knowing whether the technology holds up. Skipping the PoC because “the demo went great” is the shortcut that costs the most.

The five elements of a well-framed PoC

A proof of concept that actually decides something has five pieces, and all five are fixed before you turn anything on. If one is missing, what you have is an experiment you can always spin as a success, because you never defined what failure was.

The hypothesis: a statement you can disprove

A PoC starts with a concrete statement the test can confirm or knock down. “We believe an AI model can read the invoices that reach us as PDFs and pull the amount, the date and the supplier reliably enough to save us entering them by hand.”

Look at the shape of it. It’s a sentence that can go wrong. That’s the whole point. “We want to use AI for invoices” isn’t a hypothesis, it’s a wish, and a wish can’t be checked. If your hypothesis is written so that it’s impossible for the test to say no, you’re not experimenting, you’re looking for an excuse for what you already decided.

Real data, not toy data

This is where almost every AI PoC lies without meaning to.

The temptation is to test the model with clean examples: five well-scanned invoices, three carefully written emails, a handful of chosen cases. And of course it works. The problem is that your real data isn’t like that. It’s crooked invoices, scanned upside down, with the logo covering the amount. Emails with typos, with attachments missing, with the question buried in the third line after a long greeting.

A PoC with toy data tells you the technology works in a world that isn’t yours. The only test that decides anything is the one that uses a representative sample of your real data, mess included. If the model does well on the clean stuff and badly on the real stuff, that’s exactly what you need to know, and the sooner the better.

The success criterion: a number and a condition, decided beforehand

This is the piece most often skipped and the one that costs the most to skip.

The success criterion says what result counts as “yes, this is feasible”. And it has to be written down before you see any result, because afterward you’re no longer neutral: if you’ve put in effort, your mind will look for a way to count whatever comes out as a success.

A good criterion combines two things. A concrete measure: “the model pulls the three correct data points on at least eight out of ten invoices in the sample”. And a quality condition that protects what you can’t afford: “and it never invents an amount; at most it leaves the field blank for a person to review”. A model that gets a lot right but every so often invents a figure with total confidence can be worse than nothing, depending on what for.

The exact level is yours to set, according to what the process needs to actually save work. What doesn’t fly is not setting it and deciding at the end “by eye” whether the result convinces you.

One new concept every week

Subscribe

The deadline: short and fixed

A PoC is short on purpose. Days or a few weeks, not months. The question it answers is bounded, so the time to answer it should be too.

The fixed deadline protects against a very common ailment: the test that never ends. Without a date, a PoC that’s going so-so turns into a string of “let me tweak one more thing” that consumes a couple of people’s attention for months without ever reaching a verdict. A date set in advance forces you to decide with whatever you have on the agreed day. If in that window the technology hasn’t shown it’s feasible, that’s the answer, and it’s a useful answer.

The decision: what you do with the result

The last element is the one that gives meaning to everything above: what decision you’ll make depending on how it goes.

Before you start you have to be able to complete this sentence: “if the test goes well, we’ll do X; if it goes badly, we’ll do Y”. Usually X is moving to a pilot with real users, and Y is stopping or rethinking the idea. If you don’t know what you’d do with a “no”, it’s a sign the PoC doesn’t matter to you and you’ve already decided to press on no matter what. In that case, save yourself the test: you’re not going to listen to it.

The vendor demo is not your proof of concept

A vendor shows you their tool working beautifully. Their examples, their data, their prepared scenario. Making it look good is their job, and there’s nothing wrong with that. The mistake is taking that demo as proof the tool will work with your stuff.

Your data is different, your case has corners the seller doesn’t know about, and the demo is tuned precisely not to trip over any of them. An honest PoC is the other way around: you take the tool, give it your real data and your success criterion, and see what happens when nobody has prepared the ground. When you’re comparing vendors, this is part of what’s worth demanding from an AI SaaS before you sign: that it lets you test with your own data, not just watch its demo.

With a method versus trying to see what happens

Put together, the difference between a PoC that decides and an improvised test is clear at a glance.

	PoC with a method	Trying to see what happens
Hypothesis	A statement that can go wrong	”Let’s see if AI is any good for this”
Data	A real sample, mess and all	Clean, cherry-picked examples
Success criterion	A number and a condition, fixed beforehand	Decided at the end, on impression
Deadline	A fixed decision date	Stretches on as long as there’s enthusiasm
At the end	You decide to continue or stop	Left hanging “for later”

The right-hand column is how AI gets tested at most companies. Not out of naivety, but because building the left-hand column forces you to think through uncomfortable things before the fun part. That upfront work is exactly what separates spending money from investing it.

Before any of this, of course, you have to have picked the right case for a test. Spotting where AI adds real value and where it just makes noise is an earlier step we work through in the guide to AI use cases for companies. Here we assume you already have a candidate and want to check it out sensibly.

From the proof of concept to the decision

A PoC ends in one of two outcomes, and both are valid results.

It goes well: the hypothesis is confirmed, the model meets the success criterion with your real data. Now you know the technology is capable, and you can move on to the next question, the pilot’s, which is whether it also pays off in real work. It goes badly: the technology doesn’t get there, or gets there at a cost that makes no sense. And that, far from being a failure, is the PoC doing exactly its job, which was to save you an expensive pilot and a worse rollout on something that wasn’t going to work.

The only bad outcome of a PoC is the one that decides nothing. When by the end you’re still arguing about what “working” meant, the fault was in the framing, not the technology. That’s why the five pieces are fixed beforehand: so the result, whichever way it goes, means something.

Making these decisions with judgment, without getting carried away by enthusiasm or fear, is what we work through step by step in the AI without hype course: how to frame the test, read the result without fooling yourself, and decide the next step.

Frequently asked questions

How long should an AI proof of concept last? Short on purpose. For most cases we’re talking days or a few weeks, not months. The question it answers is bounded, so the time is too. Set the decision date before you start and keep to it even if the test is going so-so: stretching it out endlessly is the most common way for it to never decide anything.

How much does a PoC cost? Little, if you scope it well, and that’s precisely the point: to check feasibility cheaply before spending on a pilot or on production. If a proof of concept needs an investment that forces it to succeed, it’s no longer a test, it’s a bet with a conclusion written in advance.

Do I need a technical team to run an AI PoC? It depends on the case. Some tests are set up with tools that don’t require programming, and there someone from the business who understands the problem and the data well is enough. Others do call for a technical profile. What you can’t delegate to anyone technical is defining the hypothesis and the success criterion: those are business decisions.

How is a PoC different from a pilot? In the question each one answers. The PoC checks whether the technology can do the task with your data. The pilot, which comes after, checks whether that already-feasible technology pays off when you put it to work with real users over a period of time. First one, then the other.

What data should I use in the test? Yours, real, mess and all. A representative sample of what the system will actually encounter, not a selection of clean examples. If the case needs personal data to make sense, handle it carefully from the start and talk to whoever runs data protection at your company before putting it into any test.