Why AI pilots stall before production

What is this report asking, and why now?

This report asks a narrow question. A pilot has shown that a model can do a task. Why does it so often stop before the task is done in daily use, and what can a leader change before the pilot is built? It is written for the executive who sponsors a pilot, inherits one, or is about to be asked to approve one: a head of AI, a chief information or operating officer, the leader of the function whose people will use the output, and the risk, security and compliance leads who answer for what goes live.

Short answer: the model is rarely what fails. A pilot proves a model can do a task under shortcut conditions. The data is a clean extract, the users are volunteers, the run happens beside the working process, and approvals wait because it's only a test. Production asks for what the pilot never had. Data has to arrive continuously. The people who answer for risk, security and compliance have to approve. The output needs a place inside the working process, with the process changed to match. And someone has to own the system. Exhibit 1 draws these as layers beneath a working model.

Where do these needs meet?

They meet at the hand-off from the project team to the people who must run the result. The project team owns the pilot. The business function owns the process. Technology owns the platform, and risk owns the approvals. The hand-off sits in none of those jobs, so no one is assigned to carry it out, and the stall that follows is quiet. You lose the pilot budget. You also lose some of the credibility of the next proposal, which is harder to approve because the last one never arrived.

The question is getting harder to put off, for reasons that come from different directions. A pilot can start with little more than a model, an extract and a sponsor, so a business can have more than one waiting at the hand-off at the same time, each with its own project team and none with a route to production. Public audit bodies now ask for what production needs: the UK National Audit Office, reviewing the use of AI in government in March 2024, listed clear accountabilities, outcome measures, workforce impact, legacy systems and data, and the right skills among the lessons to get right at the outset (paragraph 3.8). And regulation has started to name the same things. The EU AI Act sets duties for deployers of high-risk systems in Article 26, and a regulation of 8 July 2026 moved the dates from which they apply. That gives you time to prepare, and no reason to wait.

What is a pilot actually built to prove?

A pilot answers one question quickly and cheaply: can a model do this task? Every shortcut the project team takes removes a source of delay. A clean extract saves connecting to the live system. Volunteers bring no resistance and need no training across a whole function. Running beside the process means nobody has to change it. Setting approvals aside because the work is only a test spares everyone the wait for the people who answer for risk.

Each choice makes sense for the question being asked. Together they decide what the pilot can show. Exhibit 2 sets the pilot's conditions against production's, and its last column says what each shortcut keeps out of view.

Do the public documents see the same gap?

They do. The NIST AI Risk Management Framework notes that measuring AI risk in a laboratory or a controlled environment may give useful insight before deployment, but that the measurements may differ from the risks that emerge in operational, real-world settings (section 1.2.1). It adds that accuracy figures should always be paired with realistic test sets that represent the conditions of expected use (section 3.1). The NIST Playbook, in its guidance on MEASURE 2.3, says that testing confined to test data sets doesn't directly evaluate risks and impacts in real-world environments. It suggests measuring systems before deployment in conditions like the ones expected, and documenting the differences between the measurement setting and the deployment environment.

A public-sector example shows the shape of it. The UK Government's AI Playbook describes a recommendation system for commercial agreements, which a team first built as a pilot on historical data. It was served through email marketing and in conversation, because no online platform was ready to carry it. When the platform arrived, the Playbook says, new challenges appeared: keeping recommendations flowing without a break, and measuring how people interacted with them in the service. Exhibit 3 draws the general path from a successful pilot to that point.

What does it cost you when the evidence is that narrow?

What you see is a closing report that gives results from the extract and says nothing about the conditions the pilot left out. The sponsor hears that it works. The budget conversation that follows is about scaling a result that hasn't yet met the conditions it will scale into.

The cost arrives late and all at once. The work the shortcuts postponed shows up together, just as the project team is being released and the sponsor is asking for a date. Because nobody listed the shortcuts, nobody can say how much work they postponed, and the first estimate is a guess made under pressure.

Why does nobody own the hand-off?

Each of the groups around an AI system holds a piece of its life. The project team owns the pilot until the brief ends. The business function owns the process the output would enter. Technology owns the platforms and the people who run them. Risk, security and compliance own the approvals. The move from the project team to the others sits in none of their plans, because it isn't a deliverable of the project, a budget line of the function, a service of technology or a review step of risk. It falls into the gap between reasonable jobs. Exhibit 4 shows who holds the work at each step, and where the line to the next holder is missing.

NIST describes the general condition. In its Map function it observes that people in charge of one part of the AI lifecycle often lack full visibility of, or control over, other parts, so that good intentions in one dimension can be undermined by decisions and conditions in later activities (section 5.2). Appendix A gets specific. It lists the tasks of deployment as including checking compatibility with legacy systems, ensuring regulatory compliance, managing organizational change and evaluating user experience, and it names system integrators, end users and operators among the people who carry them out. A pilot's brief is written around the model, so those tasks fall to whoever turns up when go-live is requested.

The National Audit Office found the same structure at the scale of a government. Responsibility for AI adoption sat across more than one body, with potential for overlap, and the draft strategy didn't say which of them held overall ownership and accountability for delivery (paragraphs 9 and 3.8). The UK Playbook's advice on building a team is the practical version of the same point: scaling and deploying a solution brings in software and infrastructure teams, and engaging them early gives the project clear requirements and sets a path for success.

What does the quiet stall cost?

What you see is a sequence in which everyone behaves sensibly. The project team has finished its brief and is wanted on the next one. The business saw a demonstration and assumes technology will make it real. Technology sees a model it didn't choose and has no budget to run. Risk hasn't been asked. Nobody decides to stop the pilot. It stops being reviewed, the sponsor's attention moves on, and the budget year ends.

The cost has layers. The pilot spend is lost. The knowledge the project team carried leaves with it: why the data was cleaned in a certain way, and which inputs the model handles badly. And the next proposal meets a more skeptical room, because the last one never arrived. That last cost is the hardest to find on any budget line.

Why does a model that worked on a clean extract struggle on live data?

An extract is a snapshot, chosen and cleaned by people who know the source. That work hides what production data is really like: records joined by hand, fields whose meaning changed over the years, corrections that arrive weeks late, entries made differently by different teams, and access that someone had to grant. The cleaning steps also belong to the system's performance. If the model only works after a person fixes the input, that fixing is logic, and somebody has to build it, test it and own it.

Production needs a feed where the pilot had a file. That means an owner of the source who has agreed to supply it continuously, checks where the data enters, a record of which data the model saw when it gave an answer, and a way to notice when the data changes shape. The NIST Playbook, in its guidance on MEASURE 2.4, explains why: systems meet new issues in production as the environment changes, an effect it calls drift, so that they no longer meet the assumptions of the original design. Its suggested actions include monitoring how production metrics differ from those collected in pre-deployment testing, and setting alerts for when new input data is distributed differently from the test outcomes. The framework itself lists a need for more frequent maintenance because of data, model or concept drift among the ways AI systems differ from traditional software (Appendix B).

The National Audit Office makes the same point about public bodies. It reports that significant quality problems in operational data hinder transformation, and that access to good-quality data was identified as a barrier to implementing AI (paragraphs 3.12 and 3.13). It also warns that building AI tools on top of existing systems, without dealing with the underlying issues, limits how much of the potential can be used (paragraph 3.10). For a deployer of a high-risk system, Article 26(4) of the EU Act adds a duty: to the extent the deployer controls the input data, it must be relevant and sufficiently representative for the intended purpose.

Day to day you see accuracy reported on an extract that nobody can reproduce on the live source, a refresh that is a manual task done by the one person who understands it, and a first live feed that exposes empty fields and values the model never met. The cost is time at the worst moment, or a worse outcome: a go-live with no discovery step, where wrong outputs slip quietly into records and the people downstream find them.

Why do approvals arrive after the build?

A pilot avoids approval by being declared a test. The people who answer for risk, security, privacy, legal exposure and compliance then see the design when go-live is requested, and they ask questions the design can't answer without rework. Where does the data go? What is logged? Who can see the output? How can a person override it? What happens when it fails? These are fair questions. Asked at design, each costs a decision. Asked after the build, each costs a rebuild, and every change restarts a review.

The public guidance agrees this should happen early. The UK Playbook tells teams to connect with the right assurance teams early in the project life cycle and to document review and escalation processes (Principle 10), and it states that a service must comply with the government's Secure by Design principles before it can be deployed. The National Audit Office found that public bodies named legal risk, the risk of inaccurate outputs and security risk among the barriers to implementing AI, and wanted support from the center on them (paragraphs 3.16 and 3.17).

Article 14 of the EU AI Act shows what those questions look like once they're written as design requirements. A high-risk system must be designed so that people can oversee it while it is in use. The people assigned to oversight must be enabled to understand the system's capacities and limits, to stay aware of the tendency to over-rely on its output, to interpret it, to decide not to use it or to override or reverse it, and to interrupt it through a stop button or a similar procedure that brings it to a halt in a safe state (Article 14(4)). Article 26 then makes the deployer responsible for assigning that oversight to people with competence, training and authority and the support they need (26(2)), for monitoring the system and suspending its use where it may present a risk (26(5)), for keeping the logs under its control (26(6)), and for informing workers and their representatives before a high-risk system is used at the workplace (26(7)). You can't add a stop procedure or a log in the last week of a build. They belong to the design. The NIST Playbook goes further on MAP 3.5: it says to include the relevant people in prototyping and testing, and to evaluate oversight procedures before deployment where the setting is high-stakes.

What you see is a review meeting that is the first time risk has looked at the system, then a list of questions that each change the design, with each change restarting the cycle. Reviewers who weren't involved lean toward caution, which is rational for them. The cost is months on the calendar after the builders have gone, and a lasting wariness toward AI proposals in the functions that must approve them.

What happens when the work around the model stays as it was?

A pilot shows its output in a separate view and leaves the work alone. In production the output has to be used at a defined point by a defined person, and the steps around it have to change to match. Where the model takes over a step, the neighboring steps shift: who checks, who handles the exceptions, which record is the authoritative one, what the performance measure now counts, and what the people who did the old step do instead.

When none of that happens, the old and new routes both persist. The manual route carries on because nobody has set a date on which it stops. The output turns into a rival record, kept next to the real one. People keep private copies, consult the model when they remember, and trust it when it agrees with them. The National Audit Office states the principle: adoption plans ambitious enough to deliver the benefits envisaged require changes in technology together with significant business process and workforce changes (paragraph 3.7), and it lists assessing workforce impacts among the lessons to settle at the outset (paragraph 3.8). A case study there also stresses the importance of capacity within operational teams to trial AI use cases and support adoption.

The oversight guidance covers the human side. The UK Playbook says human input must be meaningful, and advises teams to design, document and assess the stages at which human review takes place and what further information the reviewer will weigh. Article 14(4) of the EU Act names the opposite risk, a tendency to rely automatically on output. In an unchanged process both failures are likely: some people ignore the output because nothing makes them use it, and others defer to it because nothing tells them when to challenge it. On MAP 3.5, the NIST Playbook recommends differentiating the people who oversee a system from those who use it.

What you see is conflicting numbers in the same meeting, a tool used only by the people who sponsored it, exceptions with no route, and a performance measure that hasn't moved. The cost is that the gain the pilot pointed to never arrives, because the process still pays the full manual cost and now pays to review the model as well. Use declines, and the verdict that AI doesn't work here follows, though the model was never given a place to work.

Who owns the system after the project team leaves, and who can switch it off?

In a pilot the owner is the project lead, and the lead's authority ends with the brief. At go-live nobody inherits it. Ownership has separate parts that are easy to blur: the authority to decide when the system changes, the budget to run it, the duty to read its monitoring, the route by which an incident is reported and answered, and the power to stop it. A name on a slide supplies none of these. The off-switch is the part people forget, because a pilot never needed one: when it misbehaved, the team simply stopped it.

The frameworks treat this as basic. The NIST framework asks that roles, responsibilities and lines of communication for mapping, measuring and managing AI risk are documented and clear (GOVERN 2.1), that executive leadership takes responsibility for decisions about the risks of developing and deploying AI (GOVERN 2.3), that monitoring and periodic review are planned with roles defined (GOVERN 1.5), and that processes exist for decommissioning systems safely (GOVERN 1.7). Under MANAGE it asks for post-deployment monitoring plans that include appeal and override, decommissioning, incident response, recovery and change management (MANAGE 4.1), and for mechanisms, with responsibilities assigned and understood, to supersede, disengage or deactivate systems whose performance is inconsistent with their intended use (MANAGE 2.4). The Playbook's guidance on MANAGE 2.4 adds that decision thresholds for bypass or deactivation belong to continual monitoring, and that safe removal may need backup systems so work can continue.

The UK Playbook tells teams to nominate a senior responsible owner accountable for the use of AI in a specific project, to set out how the model will be maintained and managed over time with a plan for knowledge transfer, to say who has authority to change the model's code, and to have fallback processes so that critical services continue if a change is reverted or a system is terminated. The EU Act puts the same ideas into law for high-risk systems, in the stop procedure of Article 14(4)(e) and the deployer's duty in Article 26(5) to suspend use where a system may present a risk.

What you see is a live system with no name beside it in the inventory, a monitoring view nobody opens, a vendor model that can change without notice to anyone, and silence when you ask who can switch it off and what happens to the work if they do. The cost is that wrong outputs carry on, or that the system can't be removed and runs because nobody can authorize otherwise. When it's finally retired, it happens without a plan. The NIST Playbook warns that irregular or indiscriminate termination or deletion of a system can raise risk, because the system may be subject to regulatory requirements or implicated in later security or legal investigations (GOVERN 1.7).

What does a stall look like before it happens?

The signs show while the pilot is running, and none of them concerns the model. Exhibit 5 lists the signs we look for, what each usually means, and where you can check it without asking the project team to prepare anything. Read the list as questions, not a score. A single sign proves little, and signs that appear together describe a pilot whose hand-off hasn't been designed. Each one points to a person who can confirm it in a short conversation.

What is the question nobody asks?

One more sign is a question that never gets asked: what happens when the model can't be reached, or gives a wrong answer? If nobody can answer it, what you have is a pilot and no design for production. The answer isn't mainly technical. It names a person, a threshold, a fallback and a way to tell the people who used the output that it was wrong.

What changes the outcome?

Design the hand-off before the pilot is built, and not after it succeeds. That means taking these decisions at the start, each with a person who takes it and something that exists once it is taken.

First, name the owner and give them a say in the design. The sponsor takes this decision. What exists afterward is a name, a statement of what that person is responsible for and can decide, and a budget line for running the system after launch. Second, draw the route to production. The owner takes this with the project lead, technology and risk. What exists is a one-page route: where the data comes from, which systems the output enters, which approvals are required and who gives them, and what is monitored by whom. Third, change the process and not only the software, and set the date on which the manual route stops. The head of the function takes this decision. What exists is a process map with the model's place on it, and a date.

Fourth, agree what happens when the model is down or wrong. The owner takes this with technology. What exists is a fallback, the threshold at which someone acts, the person who can stop the system and the way the people who used its output are told. Fifth, agree the conditions under which the system will be retired. The owner takes this with risk. What exists is a short statement of what would end the system's life, and what is kept when it does.

The pilot itself can still take shortcuts. What changes is that each one is written down, with what it hides and when it will be removed. That list is the most useful document a pilot produces, because it is the work the hand-off will have to do. If the route turns out to have no owner willing to take the system, or no function willing to change its process, stop before you build. That costs less than a pilot that stalls.

None of this slows a pilot in any way that matters. It moves forward the questions production would ask anyway, to a stage where the answers cost a design decision and not a rebuild. The public documents ask for the same thing: on MAP 3.5 the NIST Playbook asks that the relevant people take part in prototyping and testing under scenarios similar to deployment conditions, and the National Audit Office's lessons begin with the business need and clear accountabilities. Exhibit 6 shows the order of events when the hand-off is designed first.

How well does your organization cross the hand-off?

Organizations differ in how reliably they get across the hand-off, and you can read the difference from what they can produce on request. Exhibit 7 sets out levels. At the lowest, unplanned, each pilot finds its own way and the outcome depends on who happens to care. At the next, named, an owner appears on paper. A name is cheap and reassuring, and authority isn't, so the test is whether the owner had a say in the design, holds a budget to run the system and can stop it.

At the routed level the route to production is drawn and the approvers have seen it, so the questions that would arrive after go-live have already been asked. At the rehearsed level the owner has run a change and handled a fault on live data before the project team leaves. At the standing level the hand-off is designed as a matter of course whenever a pilot is chartered, and retired systems are recorded with the reasons. The NIST framework points the same way: GOVERN 1.6 asks for mechanisms to inventory AI systems, and GOVERN 1.7 for processes to retire them safely.

To place a pilot, or a whole organization, ask for the document each level implies: the owner's written responsibilities, the one-page route, the approvers' written conditions, the record of a drill, the register of retired systems. A level you can't evidence hasn't been reached.

How do you decide whether a pilot is ready to go live?

You can make the decision with a short run of questions asked in a fixed order, and the order matters. The owner comes first, because without one nobody can answer the later questions or act on the answers. The live feed comes second, because running on real data is what exposes the design changes the approvers need to see. The sign-off of the design comes third, from the risk and security approvers and from the owner of the process that will change, since they must agree to the same design. If all the answers are yes, the pilot is ready for a go-live review, not yet for go-live.

Exhibit 8 shows the sequence. Each no has its own remedy, and the remedy isn't to push on. A pilot that fails the first question isn't at the hand-off yet. It's before it, and the first job is to name the owner.

What do we do about it, step by step?

1. 01

   List the pilots in flight

   The sponsor and the head of technology list every pilot, its stage and who will own it after launch. NIST GOVERN 1.6 asks for an inventory of AI systems, and the UK Playbook asks for a live one. You end with an inventory that has a column for the owner and a column saying whether a route to production exists.

2. 02

   Decide which pilots to carry forward

   The sponsor decides, pilot by pilot, whether to continue, hold or stop. A pilot that nobody is willing to own stops or holds. You end with a short list and the reasons written down, so a dropped pilot can be revisited.

3. 03

   Name the owner and give them authority

   The sponsor and the head of the function name one person for each pilot that continues, and agree what that person can decide, spend and stop. You end with a written statement of responsibility and a budget line for running the system.

4. 04

   Draw the route to production

   The owner, with the project lead, technology and risk, draws a single page: the data sources and who supplies each, the systems the output enters, the approvals needed, the monitoring and who reads it. You end with a route that every party has read and none has disputed.

5. 05

   Take the design to the approvers

   Security, privacy, legal and compliance review the design with the owner, before any more is built. They say what they need to see: logs, access, human review, a stop procedure. You end with written conditions for approval and a list of the changes those conditions imply.

6. 06

   Run it on the live feed

   The owner and a data engineer run the model on the live source, beside the existing process if need be, and compare its behavior with the pilot's. You end with a record of where they differ, and the monitoring that will catch drift.

7. 07

   Change the process

   The head of the function and the process owner redraw the work around the model: who acts on the output, who handles exceptions, which record is authoritative, what is measured. Everyone who will use it is trained, not only the volunteers. You end with a new process map and a date on which the manual route stops.

8. 08

   Write the run book and test the off-switch

   The owner and technology write the run book: monitoring thresholds, who acts at each, how an incident is reported and answered, how the system is stopped and restarted, what the fallback is and what would retire it. They then run a drill. You end with a run book and a record that the drill happened.

9. 09

   Hold the go-live review

   The sponsor chairs a review with the owner, the approvers and the function head. The owner presents the route, the live results, the conditions met and the fallback. You end with a signed decision to go, hold or stop.

10. 10

    Watch the first full cycle, then step back

    The project team stays through one full cycle of real use while the owner runs the system and makes the first change. Then the team is released. You end with a short review note, an owner who has run the system, and a pilot that has become a service.

How can you check a live pilot today?

• I can name the person who answers for this system after the project team leaves, and that person knows it.
• That person had a say in the design, holds the budget to run it and can stop it.
• Success is defined as a change in how work gets done, not as a demonstration.
• The model has run on the live source and not only on an extract, and the differences are written down.
• Whoever owns the data source has agreed to supply it continuously.
• Risk, security, privacy and compliance have seen the design, and their conditions are written down.
• People who didn't volunteer have used the output.
• The process map shows where the output is used and who acts on it.
• The manual route has an end date.
• There is a written answer to what happens when the model can't be reached and when it gives a wrong answer.
• Someone reads the monitoring on a set rhythm, and a threshold exists at which they act.
• Someone with authority can stop the system, and a drill has shown it can be done.
• The conditions for retiring the system are agreed.
• The day the project team hands over is in a plan, and the team stays until the owner has made a change.