The Server in the Basement Is Gone: How Fonterra Rebuilt Its Data Engine

There is a particular kind of applause a room of technology leaders gives when someone says a legacy system has finally been switched off. Fonterra’s Xanthe Sulzberger, Head of Product, and Curt Roehricht, Data Engineering Manager, earned it at the CIO Innovation Summit with a line that needed no embellishment: somebody walked down to the basement and turned that server off.

The system in question had run Fonterra’s global ingredients analytics for about a decade. Sulzberger set the scene first. Fonterra is a co-operative owned by New Zealand farming families, supplying more than 100 countries, with revenue of about $26 billion in FY25 and roughly 15,700 people worldwide. New Zealand carries the bulk of that, with 11,570 employees and 24 manufacturing sites, and about 95 percent of the milk it collects here is exported. Her team sits in global ingredients, working across demand signals, customer demand, pricing risk and supply forecasting, building the machine learning and optimisation models that decide how to get the best value from every drop of milk.

Roehricht then described what that work had been running on. The Market Analytics System, MAS, had four major parts. A custom extractor named Snaffler pulled from more than 130 source profiles by API, web scrape or database. SSIS ran the transformations, stripping out and reshaping through enormous jobs. An on-premise SQL Server from 2019 held more than 300 tables across multiple databases. Power BI sat on top for the reports and dashboards. Around the edges, operational systems pulled from it, teams built applications on it, and analysts ran machine learning models on their own laptops.

"It turns out there is a limit to how much memory you can throw into a physical server, or how much CPU you can throw at it."

The failure list will be familiar to anyone who has inherited a critical system. When something broke, you hunted through it to find where. Maintenance overhead ran to half the team’s time or more, a figure Roehricht said was echoed at a keynote he attended in San Francisco, where the claim was that most companies still lose more than 50 percent of their data effort to upkeep. Scaling meant physical limits, and his first weekend on call was spent with Microsoft trying to squeeze more out of jobs that were failing overnight. The advice was to spread the jobs out. Governance was thin. Hard-coded values sat in the code, some unchanged for ten years or more. New engineers took months to understand how the pieces connected, and there was no documentation, because the system had been built by vendors who then left. Even without a migration, the underlying tools were reaching end of support, so upgrades were coming anyway, and an upgrade is usually a migration wearing a different hat.

An assessment in 2024 concluded the system had to go. Fonterra had already begun building its data platform on Databricks, so that became the destination. The project kicked off in 2025, everything was on the new platform by the middle of the year, both systems ran side by side for a period so results could be tested, and the old one was retired a few weeks before the Summit. It came in under budget and under time.

The most instructive decision was one they made deliberately at the start. A year can look slow for a migration, and they chose it anyway, because they refused to lift and shift.

"We rebuilt it from the ground up. Where we had been scraping, could we now get an API? Where we had assumed this number needs to be multiplied by that, is that real? Or do we even need that data?"

That choice had a cost he was candid about. More than once he had to tell the business that a number they had relied on for five years was about to change, because it had been wrong. Rebuilding surfaces those inheritances rather than carrying them quietly forward.

Then came the part that turned a good migration into a story worth telling. Technically the project was a success, but the people using the data still had the same day to day experience. They still asked the reporting team for a query. They still waited for someone to write a report. They still exported to Excel and did their own manipulations, then wrote the commentary by hand. The models were on the new platform, but the question of how to put them in users’ hands was unanswered.

Fonterra’s answer came from looking sideways at what other teams in the business were already doing with Genie spaces, Databricks’ natural language interface that sits on top of governed data and lets people interrogate it directly. A technical business analyst sat down with a business user, and two weeks later the first space was live. Users loved it. Within another fortnight a second team had one.

The feedback Sulzberger and Roehricht went back and collected is the part worth dwelling on. One user called it their new best friend at work, using it almost every day. For a non-technical business user, that is the whole ballgame. But the most powerful comment was about curiosity. Previously, a hunch about how one thing correlated with another meant a long chase through multiple systems to validate it, so people either abandoned the question or sank days into it. Now the hunch could be tested in minutes, and analysts found relationships in the data that had not been obvious from reading the tables.

Roehricht was careful about what this does and does not replace. A Genie space will not know that a decision made on the other side of the world last week is about to move a market. The analysts know that. He described it as another coworker, one that offers a starting point and supports deeper analysis rather than substituting for judgement.

Their key lessons were refreshingly unglamorous. Target specific use cases, and go looking for the loudest pain point, the manual step someone repeats over and over. Co-design with the business and let them drive evaluation, because they know their data better than the engineers ever will and they will spot a wrong answer first. Get something into users’ hands early and iterate, rather than polishing something complete and discovering it was not what they wanted. Acknowledge that AI can be wrong, while noting the false comparison hiding in that worry: this project proved that reports trusted for ten years were also wrong. Humans make mistakes too, so benchmark and be honest about both.

Two more stood out. Reuse other teams’ work, which is what let them move in fortnights rather than quarters, and it is also why conferences like this one pay for themselves. And monitor usage, because most organisations have thousands of dashboards where only a handful are ever opened. Roehricht checks usage almost daily and is happy to throw something away and start again, because rebuilding now takes days rather than months.

Where it goes next is a simple ambition. Whatever department you are in, you should be able to ask a question in plain language and have your data answer it. With the dashboards, the apps and the Genie spaces already in place, Fonterra’s remaining work is tying them together. For a co-operative whose job is getting the best value out of every drop of milk, being able to ask better questions faster is not a technology outcome. It is a farmer outcome.