Esc
← All Writing

AI Audit: The 3 Questions I Ask Before Any AI Build

Most AI shortlists start with the tool. Run every idea through A.R.T., Applicability, Readiness and Transformation, before anyone writes code.

3 questions I ask before any AI build. And none of them is “can AI do this?”

Nowadays most AI shortlists start with the tool. Someone sees a demo, someone reads about agents, a pilot gets funded. Six months later the board reads the MIT report that 95% of generative AI pilots are failing and asks the obvious question: why are ours stalled, and where is the return?

It is not only GenAI. Cisco’s 2026 industrial AI study found 61% of industrial users deploying physical AI, but only 20% have scaled it. One of the two impediments they name first is identifying the right problem. That is not a model problem. That is an audit problem.

The question was never “can AI do this?” It’s which AI, when, where, and how. So I run every potential solution through the A.R.T. funnel, in this order.

What is the A.R.T. framework?

A.R.T. stands for Applicability, Readiness, Transformation. It is the first thing I do in any AI audit, and it distils the use cases that can actually pay back.

  • A, Applicability: is AI even the right solution?
  • R, Readiness: is the data ready now, and will it be in a year?
  • T, Transformation: what changes on the P&L, in an ROI number someone signs off on?
About 20 AI ideas flow through three filters. Applicability drops those too complex or too naive, Readiness drops those the data or systems can't support, Transformation drops those without a workable ROI or timeline, and about 4 reach the shortlist of profitable initiatives.Many ideas in, a short list out. The order matters.About 20 AI ideas flow through three filters. Applicability drops those too complex or too naive, Readiness drops those the data or systems can't support, Transformation drops those without a workable ROI or timeline, and about 4 reach the shortlist of profitable initiatives.Many ideas in, a short list out. The order matters.

Scroll sideways to explore the full diagram.

The order matters. There is no point checking data for a problem that a rule can solve, and no point building an ROI case for something your data cannot support yet. Each filter saves you the cost of the next one.

A, Applicability: is AI even the right solution?

Start with the goal, not the tool. Break the goal into 5 to 9 smaller decisions. Then go through them one by one. If a rule works, use a rule. Only use AI when rules break.

This sounds obvious. It isn’t, because the shortlist usually arrives with the answer already in it: “we need an agent”, “we need a chatbot”, “we need GenAI for this”. A GenAI model is a plier. Very useful, and the wrong tool for most of the jobs on a factory floor or in a planning team. Many times the work needs a forecast, an optimiser, a classifier, or plain automation rather than an agent.

Then name the logic or profit leak. Where is the P&L bleeding? Which decision, made by whom, is costing money today?

No logic leak, no use case. You have a tech demo.

I check the leak against a simple equation: profit is revenue minus cost, given customer satisfaction, efficiency and governance. Every use case has to attach to at least one of those pillars. My personal view, at least two. Otherwise sooner or later the ROI will not be there.

Where was the leak when lead predictions got the blame?

A premium auto distributor wanted AI to predict which leads would buy, yes or no. About 4 buyers for every 1,000 leads. The obvious proposal was a classification model.

But the buying journey was not one yes/no decision. A buyer moved from a marketing touchpoint to a first call, a test drive, a sales consultation, a finance discussion. At each step new information appeared, and at each step the salesperson could do something about it.

The leak was what happened at each stage, but the predictions were getting the blame. So we stopped asking “will this customer buy?” once, at the start. We guided each stage instead: what to talk about now, which vehicle and finance option fits, what to prepare for the next step.

Result, a 5x lead conversion.

Same data, same people. The difference was asking where the leak was before choosing the model.

R, Readiness: is the data ready now, and will it be in a year?

Data is part of every AI audit I do, because without data, AI doesn’t exist. Readiness asks three things about the data:

  1. History: do you have enough past outcomes, logged next to the inputs that led to them?
  2. Granularity: is the data recorded at the level the decision is made, per SKU, per store, per machine, per shift? Or only as a monthly summary?
  3. Access: can the model get the data when the decision is made, or does someone export a CSV every Friday?

Then two things about the team: skills and bandwidth. Who will own this after the pilot? Who checks the output on a bad day?

A red here doesn’t mean stop. It’s your roadmap before anyone writes code. It tells you what to fix first, and that is often cheaper than the model.

And if you score yourself green on everything, you are probably lying to yourself about your data.

Why can’t one accuracy number cover an 18-month forecast?

Because accuracy is set by the inputs you have at each horizon, not by the model.

The clearest readiness lesson I’ve had was in a boardroom, not in the code.

At L’Oréal (through Nagarro) we were building promotion-aware demand and procurement planning across 5 markets and 8,044 SKUs, on three horizons: daily for the next 3 months, weekly for the 3 months after, and monthly out to about 18 months.

The board expected one high accuracy number across all 18 months. So I took the marker and went to the board, and walked through which inputs actually exist at each horizon.

  • Short horizon: planned discount depth, promotions, vouchers. Rich inputs.
  • Middle horizon: planned inventory levels and marketing expectations. Fewer.
  • Long horizon: nothing. Just the trend of the product.
Three forecast horizons on one time axis. Daily for the next three months has discount depth, promotions and vouchers as inputs; weekly for months four to six has planned inventory and marketing plans; monthly out to about eighteen months has only the product trend. The inputs, and so the achievable accuracy, thin out as the horizon gets longer.Accuracy is set by the inputs you have at each horizon, not by the model.Three forecast horizons on one time axis. Daily for the next three months has discount depth, promotions and vouchers as inputs; weekly for months four to six has planned inventory and marketing plans; monthly out to about eighteen months has only the product trend. The inputs, and so the achievable accuracy, thin out as the horizon gets longer.Accuracy is set by the inputs you have at each horizon, not by the model.

Scroll sideways to explore the full diagram.

With those inputs, one number for 18 months was simply not possible. Once they saw it in their own words, they understood everything, and we agreed realistic targets per horizon before building anything.

That is readiness. Accuracy is set by the inputs you have at each horizon, not by the model. And it is why I set accuracy and limits up front, before the build, not after the first disappointing demo.

T, Transformation: what AI ROI shows up on the P&L?

The last filter is the one most pilots skip. What changes on the P&L, in an ROI number someone signs off on?

Is it making an analyst 10% faster, or changing how a decision gets made? Those are not the same thing. A faster analyst is a nice slide. Unless that time turns into money, removed cost or avoided hiring, it does not show up anywhere a CFO looks. I made the same point about measuring the whole process, not the first draft, and it holds for every kind of AI.

In my work the number usually lands in one of these places:

Where it landsWhat it looks likeOne I have seen
Cost removedFewer vendors, fewer licences, less reworkA materials-science manufacturer went from 7,000 to 5,000 vendors, about $100K a month saved
Working capital releasedLess stock held for the same serviceAscena replenishment across 2,800+ stores, about $150K a month of net inventory saved
Margin addedLess product wasted or lost to claimsA fruit exporter predicting shelf life per batch, 10%+ margin
Capacity gainedSame team, more volume15 analyst hours a week freed in the vendor work

If the ROI isn’t clear, back to the drawing board. Not “we’ll measure it after go-live”. Back to the drawing board.

How do you run an AI audit on your own AI list?

Write each idea as a decision, run it through the three filters in order, and rank what survives.

You don’t need me for the first pass. Take the list you already have, every idea, pilot and vendor pitch, and do this:

  1. Write each idea as a decision. “Predict which leads buy” becomes “decide what the salesperson does next at each stage”. If you can’t write the decision, park the idea.
  2. Applicability. Break it into 5 to 9 steps. Mark which steps a rule can handle. Name the leak in money. No leak, cross it out.
  3. Readiness. Score history, granularity, access, skills and bandwidth as green, amber or red. Reds become the fix list, not the kill list.
  4. Transformation. Write the P&L line it moves and the person who will sign off the number. No owner, no number, back to the drawing board.
  5. Rank what survives by size of the leak and how few reds stand in the way.
FilterThe questionKill signalIf it fails
ApplicabilityIs AI the right solution?A rule works, or no leak can be namedWrite the rule, or drop it
ReadinessIs the data and team ready now, and in a year?Red on history or access with no plan to fixFix the data first, it’s the roadmap
TransformationWhat changes on the P&L?No number, no ownerBack to the drawing board

What survives all three is the short list worth building. For one department, it is usually the top 3 or 4 use cases that can yield ROI in a sizable time frame. The rest need more data maturity first. That’s not a rule of thumb, it’s what happens in practice, because in most places the data is not in proper shape. It needs capturing first, and only then is it worth building.

Mark everything else as a science fair project, the kind born from pressure and FOMO instead of pain, and don’t tell MIT or Gartner about it.

When should an AI audit say don’t build it?

When the process doesn’t make sense in practice yet, or when the extra effort won’t yield any ROI.

If it’s not worth doing, I’ll tell you, and that happens more often than you’d think. AI is a tool, not a magic wand. Half the time the answer is a better process. Two real examples.

Should AI agents promise stock from shared inventory?

For a trading company we were building an agentic CRM for lead management. The first ask was simple: update the leads, their qualification and their needs.

Later the owner asked me to bring the inventory in directly, so he could see which inventory was going to whom. In an ideal scenario, agents can do this. They can do the parsing and all the stuff.

The problem starts here. What if a product is already blocked for one customer, and a different lead asks for the same material? Two agents, one stock, two promises.

So as an interim, and as a best practice, I recommended an end-of-day sheet. All the blocked materials get calculated at the end of the day. Only what is available for consumption the next day is used to check whether you have the quantity to fulfil a new requirement.

The ask was genuine, and a full solution was possible. But the process didn’t make sense in practice yet, and building it then would have ruined the reliability of the whole system. That is Applicability and Readiness doing their job.

Which products don’t need a more accurate forecast?

In demand forecasting and inventory management, a lot of times people want too much accuracy on the wrong products.

Think of it in ABC-XYZ terms. A is the most sellable, C the least. X is the most predictable, Z the least. Out of a look for perfection, teams want the CX and CY products perfected. But optimising those doesn’t guarantee any yield, because they have neither the volume nor the sales to make it worth it. A simpler model, usually a mean, works better there.

The focus should be on the A, B, X, Y combination. That’s the holy part. If you put more effort into the rest, it won’t yield any ROI. That is Transformation saying no.

Why do AI pilots stall before the build even starts?

Because which problem, which tool, which data and which number get decided before the build, and that is where most pilots go wrong.

The interest is not the problem. Capgemini’s 2026 research found 79% of large organisations engaging with physical AI, and only 27% deploying or scaling it. The money is being spent. The gap is between a pilot that works in a demo and one that runs on your floor, on your data, every day.

That gap is mostly decided before the build. Which problem, which tool, which data, which number. A.R.T. is just a way to answer those four in the right order.

So take your current AI list and run it through these three. How many are still standing? And for the ones that aren’t, what would it cost you to find that out after the pilot instead of before?

Frequently asked questions

What is an AI audit?

An AI audit checks each AI idea before money is spent on it. I run every idea through three questions in order: Applicability (is AI the right solution, or will a rule do?), Readiness (is the data and the team ready now, and in a year?) and Transformation (what changes on the P&L, in a number someone signs off on?). What survives all three is the short list worth building.

What does A.R.T. stand for?

Applicability, Readiness and Transformation. Applicability asks whether AI is needed at all and where the logic or profit leak is. Readiness asks whether the data, skills and bandwidth exist. Transformation asks what changes on the P&L.

How do I know if a problem needs AI or just rules?

Break the goal into 5 to 9 smaller decisions. If a rule works for a step, use the rule. Bring AI in only where rules break, and only if you can name the logic or profit leak it fixes.

What does a red AI readiness score mean?

It does not mean stop. A red on readiness is your roadmap: it tells you which data, access or skills to fix before anyone writes code. A team that scores green on everything is usually not looking hard enough at its data.

How should AI ROI be measured?

As a change on the P&L that a finance owner signs off on: cost removed, working capital released, revenue added or capacity gained without new hires. Making an analyst 10% faster is not ROI unless that time turns into money.

Is this where your AI is leaking?

A 45-minute diagnostic shows where the logic between your data and your decisions breaks. No pitch, no deck.

Book a 45-minute diagnostic Or see how the one-week AI Audit works

The Weekly AI Decision Brief

One operating question a week.

Email only · the weekly operator brief. Prefer the community feed? Read on Substack.