The models are not your advantage: Joseph Sullivan on the data under AI

Joseph Sullivan opened with a show of hands. Who in the room had been asked by a board member or executive committee about their company’s AI strategy? Almost every hand went up. Keep your hand up, he said, if that question was followed by: what is our data strategy to support the AI strategy?

Plenty of hands stayed up, which he said was about the best response he had seen. Normally most of them drop, because at executive level the focus is on the models, the agents, the autonomous use cases and the outcomes, “all the sexy parts of AI”, and very little on the foundations that make those outcomes possible.

Sullivan is Regional Vice President and Head of Specialist Sales for Asia Pacific and Japan at Informatica from Salesforce. His session at CDAIO New Zealand, “Beyond the Hype: How to Build the Foundation that Powers AI Success”, followed NZQA’s account of scaling AI marking and was about those foundations. His closing line is the one to remember: everyone in the room has access to the same models.

TL;DR

• Sullivan cited research that almost nine in 10 AI pilots never scale into production, usually not because they failed but because the data underneath was not ready.
• He quoted Gartner’s finding that 85 percent of AI projects fail because of poor data quality, against only 7 percent of organisations that believe their data is ready for AI.
• Agents raise the stakes: bad data that a human once caught at human speed now compounds across hundreds or thousands of machine decisions.
• Correct data is not enough. AI needs context, because marketing, sales and finance can each give a different, correct answer to “how many new customers did we add last quarter?”
• Informatica’s framework has three parts: unlock the data, trust it, and activate it.
• Using AI to build the data foundation fails on accuracy and cost. In a McKnight Consulting Group benchmark he presented, ungoverned access burned 20 to 30 times more tokens and returned far less accurate answers than a governed metadata layer.

The building nobody looks at

Sullivan is an Australian, and a Victorian, who has spent the best part of 20 years outside the country, the last 10 in Hong Kong. That is where he became interested in skyscrapers. We look at the grand glass and steel monuments, he said, rising 50, 60, 70 storeys or more. We rarely think about what sits underneath, holding them up.

His example was the new JPMorgan Chase headquarters in New York, and the parallels he drew with organisations trying to become AI enterprises. The bank had to deal with a huge amount of legacy. An existing building had to be knocked down. Train lines run underneath the site, along with sewers and other infrastructure, and some of the old foundations had to be left in place and built on top of.

That, he said, is what organisations face with legacy databases, legacy systems, poor data quality and poor data governance as they move towards becoming an AI enterprise.

AI first or data first

He sees two general approaches in the market. AI first means getting moving with whatever data you can get your hands on and trying to show outcomes. That can lead to unfortunate results, because the foundations may not hold. Data first means putting the data foundations in place alongside the AI as you mature: the right data quality, and a single trusted understanding of the business.

Organisations that go AI first often meet what he called the AI pilot paradox: very successful pilots that never make it into production. Research has shown that almost nine out of 10 AI pilots never scale into production, he said, and not because they failed. They usually produce good outcomes and prove the use case. They just cannot scale.

Why? He quoted Gartner: 85 percent of all AI projects fail because of poor data quality. Set that against the money being spent on AI, and the finding that only 7 percent of organisations believe their data is ready for it, and something is going wrong.

From human speed to machine speed

AI has been around a long time, he reminded the room. People were studying AI languages when he was at university nearly 30 years ago. Progress was slow until about five years ago, when it exploded with the first generative pre-trained transformers.

In that first era there was a human in the loop. If the underlying data was wrong and the answers came back wrong, a human could usually tell. We could sense check it, intercept the bad data, and even if we made decisions on it, we were moving at human speed, so there was only so much damage that could be done.

With agents and autonomous AI, machines are moving at machine speed. Agents talk to agents, and teams of agents work together. A wrong decision made on poor data compounds quickly, across hundreds if not thousands of decisions, and rolling that back in the business can be very difficult.

Correct is not the same as right

Even correct data is not enough, he said. Ask “how many new customers did we onboard last quarter?” and, depending on which part of the business you ask, you get several answers that are all correct. Marketing, sales and finance each define a new customer differently. Modern guardrails catch obviously wrong answers. These are all right, so the AI picks one and makes decisions on it.

The data is not technically wrong. The AI did not have the context of the question being asked.

"Context is what makes the data meaningful."

Context means what the data is about, where it came from, its history, how it is used and what it relates to. That full context is what allows AI to understand the business as people understand it.

His example was a customer service agent built to quote renewal pricing. It has access to previous contracts and pricing history, so it offers the standard price with the usual annual uplift. What it does not know is what the service and support teams know: this customer has had a bad run, with outages, economic headwinds and real unhappiness. A person would offer a discount to keep them. The agent offers the uplift, and the customer leaves.

Unlock, trust, activate

Building that context is hard. Data is scattered across tens or hundreds of systems, from mainframes and AS/400s to modern cloud platforms. He described an organisation in Perth that, looking for personal information, found it in 16 tier-one systems and 153 systems in total. Creating a unified understanding of a customer across all of that is difficult, doing it in real time is harder, and trusting it is harder still.

Informatica’s framework for building that contextual understanding has three parts, and he stressed it is a way of mapping what you already have rather than a prescription.

• Unlock. Connect systems wherever they live, from mainframe to modern cloud, in real time. Integrate them. Then understand the data, using AI to recognise that “column one” in a legacy table and “first name” in a modern system are the same thing, and linking business and technical metadata.
• Trust. Validate the data and catalogue it. Protect it, with policies that follow the data and govern how both people and AI can access and use it. Unify it into a single trusted understanding of customers, suppliers, locations or assets.
• Activate. Push that trusted data into the tools and the AI that make decisions with it.

Can we just throw AI at it?

Normally, he said, he would stop there. But he keeps getting the same question: we understand the need for a data foundation, but we don’t have the time or the money, and the executive team wants to move faster. Can we just use AI to build it?

It does not work, for two reasons: accuracy and cost. Using AI to build your data foundation, he said, is like building a skyscraper and getting a general handyman or the work experience kid to do the foundations.

On accuracy: does your AI understand how your business defines things, how it operates, which systems can be trusted? Typically not. His example came from travel. Ask an AI whether you need a visa to visit the United States, and an answer from an unverified Reddit thread, however popular, deserves less trust than one from the US Embassy. The Reddit answer is not even wrong, he said; you do not need a visa, but you do need an ESTA, and without one you will not get into the country. That governed, trusted understanding has to come from your business users. AI cannot build it.

On cost, he pointed to a token-maximising phase over the past year or so, when organisations told everyone to use AI for everything, followed by profit warnings on earnings calls.

His slide listed Uber burning through its entire 2026 AI budget in four months, and an unnamed Fortune 500 company that rolled Claude out across the business with no guardrails and ran up a $500 million bill in a single month. Some of it is using expensive models for jobs a cheaper model could do. Some of it is using AI for tasks it is not set up for, and data management is one of them.

He gave two examples. A global brand found that people using natural language interfaces to move data from A to B had spent more than US$500,000 on work that can be done for pennies. And a large bank’s asset management arm, deploying AI models on a data platform, found itself spending about a million Japanese yen a day. After it deployed data governance tools, put policies in place and curated its data, its AI costs dropped by 91 percent.

Token bleed

The reason, he said, is a phenomenon now called token bleed. When an AI model works through a large dataset, it burns tokens building an understanding of the data before it can make a decision. It does not remember that understanding, so it does it again for the next question, and the next.

To put numbers on it, Informatica worked with McKnight Consulting Group, an independent analyst, on a benchmark. It tested one step of an agent’s work, discovery, with a simple prompt: show me all the assets needed to build a report that involves a government ID. The test data held known right answers, real-world issues, and decoys such as other identifiers that were not the government ID. Every test used the same Claude Opus model, and only the way it reached the data changed:

• Direct database access: a live connection to the data across Snowflake, Oracle and S3.
• Context stuffing: exporting the metadata structure into the model’s context.
• A governed metadata layer: the data scanned into Informatica’s catalogue with the default implementation, no curation, made available to the agent through MCP.

At 1.5 million objects, the governed layer answered in 2,112 tokens. Direct database access and context stuffing used between 20 and 30 times as many. On accuracy, the governed layer scored a perfect F1 score: every correct result returned, nothing missed, nothing false. The other two approaches scored, in his words, about the same as a coin toss, missing correct results and returning thousands of incorrect ones for the AI to make decisions on.

Multiply that across 10,000 agent calls a day, which he said is not out of the question for a mid-sized organisation, and he put the difference from a governed approach at about $2.7 million.

The same models for everyone

He finished with the point he wanted the room to keep.

"Everyone in this room can have access to the same model. The models are not your advantage."

The advantage, he said, is in how you activate your data and its context. The organisations that win will be the ones that can unlock their data, trust it and activate it. As he left the stage, the chair noted that accuracy and cost would be the thread the next panel picked up.