From idea to AI pilot: how to test it without betting the company

Before you test AI in your company, answer one question: what has to happen for you to say it went well? If you cannot answer in a concrete, measurable sentence, you are not ready for the pilot yet. You are ready to think a bit more.

Most AI projects that fizzle out do not fail because of the technology. They fail because nobody defined what winning meant. It starts with enthusiasm, a couple of demos that impress in a meeting, and six months later nobody can say whether it was worth it. The money was spent, the team got tired, and the conclusion is a shrug. A well-built pilot exists precisely so that does not happen.

What an AI pilot is (and what it is not)

A pilot is a small experiment with clear limits, meant to answer a business question before you commit real time and budget. One use case, one team, one closed time window. Nothing more.

It is not a company-wide rollout dressed up as a test. Nor is it a toy that a couple of curious people use now and then to “see how it goes”. Those two things get confused with a pilot all the time, and both waste money: the premature rollout because you expose customers and data before knowing whether it works, and the toy because it burns hours without anyone being accountable for anything.

The difference lies in four decisions made before you switch anything on. What you will measure. When you will stop. How far it reaches. Who answers for each risk. If those four are not on the table on day one, you do not have a pilot. You have a bet with someone else’s budget.

Even before building it, it pays to have chosen the case well. That part, spotting where AI adds real value and where it just makes noise, is what we work through in the guide to AI use cases for companies. Here we assume you already have a candidate and want to test it sensibly.

The success metric is decided before, not after

This is the piece almost everyone skips, and the one that costs the most.

Picture an agency that takes too long to send quotes to its clients. Each request arrives by email, someone reads it, looks up prices, drafts and replies. The goal of the pilot is not “use AI for the quotes”. That goal cannot be measured, so you will never know whether you met it. The goal is something like “cut the average time between a request arriving and a quote going out, from two days to half a day, without complaints about errors going up”.

Look at what that sentence has. A single measure. A starting point you already know. A number you want to reach. And a condition that protects quality, because going faster while getting it wrong is not winning.

A useful success metric meets several conditions at once: you can measure it with data you already have or can easily collect, it reflects a real business benefit rather than technical excitement, and you cannot “eyeball” it at the end. If, when the pilot ends, the only way to know whether it worked is to ask people if they liked it, then there never was a real metric, just an impression.

Pick one. Just one as the main metric. When a pilot chases five goals, it really chases none, because you will always be able to point to “this one did improve” and hide the other four.

The stopping criterion: when to switch it off

If the metric says what winning is, the stopping criterion says when you stop playing. And it has to be set beforehand, while nobody is yet excited or hurt by the results.

A stopping criterion has two halves. One is the deadline: this pilot runs six weeks, not one more, and on that date we make a decision with what we have. The other is the failure threshold: if halfway through the improvement is not even close to what was expected, or a serious problem with data or customers shows up, you stop earlier.

Without this the zombie pilot appears. The one that works neither well enough to roll out nor badly enough for anyone to dare kill it. It sits in limbo consuming the attention of a couple of people for months, always “almost there”, always one more improvement away from convincing someone. A deadline written in advance takes away that pilot’s chance to drag on forever, because when the day comes you have to decide yes or no with whatever data you have.

Putting an expiry date on something you have invested in is hard. That is why you decide it before investing, while it is still easy.

One new concept every week

Subscribe

Limited scope: small on purpose

A pilot is made small on purpose. Small is what you can control, measure and, if needed, switch off without wreckage.

Bounding the scope means making four very concrete decisions:

• A single case. The quotes, not “customer service” as a whole. If the case is broad, carve off a slice and test that slice.
• A small team. The people who will actually do the work with the tool, not the whole staff. Five people using it for real teach you more than fifty watching from a distance.
• Low-risk data first. Start with information that exposes no one if something goes wrong. Feeding customers’ personal data into the first day of a test is asking for the kind of problem that gets expensive and ends up in court.
• A short time window. Weeks, not quarters. The longer the pilot, the more it looks like a covert rollout and the less like an experiment.

Starting small protects two things a decision-maker does not want damaged: the budget, because a bounded test costs little and fails cheap, and the reputation, because if the experiment goes wrong, it goes wrong in a corner and not in front of your customers. The temptation of “since we’re setting it up, let’s try it on everything” is real and has to be resisted. The big version comes later, if the pilot says it is worth it.

Here it is worth not confusing the pilot with the phase that sometimes precedes it. Checking that the technology can do the task at all is a proof of concept: it answers “can this be done?”. The pilot answers a different question, “is this worth it in real work?”. They are different things, and mixing them leads to measuring badly.

Mitigations with names attached

Every AI pilot has risks. The difference between a serious one and a reckless one is not that the reckless one has risks and the serious one does not. It is that in the serious one every risk has a named owner, and in the reckless one every risk is handled with a “we’ll see”.

The usual risks of an AI pilot are easy to anticipate. The tool can give a wrong answer with total confidence, and if that reaches a customer without anyone reviewing it, you have a problem. It can touch personal data covered by European data protection, and there a slip can be illegal, on top of costly. You can end up tied to a single provider whose price or terms change once you already depend on it.

Mitigation is not a wish list. It is assigning each of those risks to a person who answers for it during the pilot. Who reviews the answers before they go out to the customer. Who decides which data goes in and which does not. Who watches that spending does not blow up. A risk without an owner is a risk nobody is watching.

Prioritizing which cases deserve this effort, and which should not even reach a pilot given their mix of risk and impact, is a task in itself. You can order it with a risk and impact matrix before deciding where you invest the first test.

A note on the legal side. None of this is legal advice. If your pilot is going to touch personal data, talk to whoever handles data protection at your company before you start, not after.

A sensible pilot versus a blind test

Put it all together and the difference between the two ways of testing AI is clear at a glance.

	Sensible pilot	Blind test
Success metric	A concrete measure, set beforehand	”Let’s see if something improves”
Deadline	A closed decision date	Drags on while there is enthusiasm
Scope	One case, one team, low-risk data	”Let’s try it on everything”
Risks	Each one with an owner	”We’ll see”
At the end	Decided with data	Decided on the boss’s gut feeling

The right-hand column is how AI gets tested in most companies. Not because people are naive, but because building the left-hand column forces you to think about uncomfortable things before the fun part. That is exactly the work that separates a cost from an investment.

From the pilot to the decision

A pilot ends in one of three decisions, and all three are legitimate outcomes.

Scale, when the metric was met and the risks were controlled: then you widen the scope with what you have learned. Adjust, when there were good signals but the metric fell short: you change one piece and test again, with a new pilot and its own criterion. Or stop, when it was not worth it. Stopping in time is the pilot doing exactly its job, which was to save you the cost of a rollout that was not worth the trouble.

The quietest mistake is using the pilot to justify what you had already decided to do. If you are going to scale no matter what, there is no real pilot: you are just looking for a photo for the meeting. A pilot is only worth something if it genuinely could have come out no.

Making these decisions with judgment, without falling for the hype or the fear, is what we work through step by step in the AI without hype course: how to choose the case, build the pilot and read the result without fooling yourself.

Frequently asked questions

How long should an AI pilot last? Long enough to gather sufficient data and short enough not to turn into a covert rollout. For most cases we are talking weeks, not quarters. The rule is to set the decision date before you start and stick to it.

Do I need a technical team to run a pilot? It depends on the case. Many tests are set up today with tools that require no programming, and for those it is enough to have someone from the business who understands the problem well. Others do need a technical profile. What you cannot delegate to anyone technical is defining the success metric and the stopping criterion: those are business decisions.

What budget is reasonable? The one you can lose without it hurting if the pilot says no. That is the advantage of bounding the scope: a small test fails cheap. If a pilot needs an investment that forces it to succeed, it is no longer a pilot, it is a bet.

How do I know whether to scale, adjust or stop? By comparing the result against the metric and the stopping criterion you set at the start. If you defined them well, the decision almost makes itself. If you find yourself arguing over what “success” meant at the end of the pilot, the problem was at the start, not in the result.

Can I use real customer data in the pilot? With great care and almost never at the start. Begin with data that exposes no one. If the case needs personal data to make sense, talk first to whoever handles data protection at your company and treat that risk as a mitigation with an owner, like any other.