We thought we were scaling AI. We were scaling NZQA

One hundred thousand essays, marked in about 54 minutes. For most organisations that would be the end of the story. For the New Zealand Qualifications Authority it turned out to be the start of a much bigger one.

Bill Moses, NZQA’s Chief Information Officer, and Vipin Kumar, its Chief Customer Officer, took the stage at CDAIO New Zealand straight after Te Aroha Grace’s mihi whakatau on 19 October. Their session, “Same Fundamentals, New Scale: What Happens When AI Actually Works?”, was the first keynote of the conference.

Their argument was simple to state and hard to live. Once AI proves it can do the task, the work stops being about the technology. It becomes integration, governance, security, operating models and an organisation ready to run it. And whether AI arrives as a product you build or a tool your people use, it ends up needing the same foundations.

TL;DR

• NZQA used to take about 10 weeks to release literacy and numeracy results, leaving learners and teachers almost no time to act before the next assessment event.
• A two-person data science team built an AI marking model on DeBERTa, a small language model of about 300 million parameters, trained on around 30,000 essays already marked by experts.
• In live production it marked 100,000 essays in about 54 minutes and brought results forward by four and a half weeks. Human expert markers still mark the responses around the pass mark.
• Proving AI can do a task and running it as a trustworthy service are two very different things: production needed architecture, integration, security, operational design and controls, with assessment experts at every stage.
• NZQA’s AI as a product (marking, a question-answering chatbot, overseas qualification assessment) and AI as a tool (AI-supported software engineering, Copilot) converged on the same capabilities.
• The lesson they closed on: “We thought that we were scaling AI. But we were wrong. We were actually scaling NZQA.”