Robodebt fears as Law Commission reviews automated decision making

Mahvash Ikram and Neil Sands

The Law Commission is set to review legal challenges posed by the government’s use of automated decision making (ADM), although there are concerns the technology is already becoming embedded in state bureaucracies without adequate safeguards.

ADM, which allows decisions to be made without human input, is not new. But its use is set to be turbocharged by artificial intelligence (AI), particularly after Finance Minister Nicola Willis said increased adoption of AI will help the government cut almost 9,000 public sector jobs.

Law Commission president Dr Mark Hickford said there are at least 19 pieces of legislation that allow various government agencies to use ADM. He said this fragmentation had created a “siloed approach” across the public service which did necessarily take into account the legal questions that ADM poses.

“Most of the emphasis around digital has been on efficiency and productivity gains, but we also have to think about citizens and rule-of-law values and so forth, just [to] give assurance and certainty or guidance to those interacting with these new technologies that various rights will be protected,” he said.

The commission’s general counsel Linda McIver said key stakeholders such as the Ombudsman, Community Law and the New Zealand Law Society (NZLS) had all recently highlighted the need to investigate legal challenges arising from the use of ADM.

“This issue seems to be attracting a lot of attention at the moment because the government has encouraged the public service to examine artificial intelligence as a means of making decisions and carrying out government functions… but there seems to be a lack of legal focus on some of these items… or how governments should use this new technology to make decisions that are compliant with the law,” she said.

Hickford said the commission’s review will focus on creating a coherent framework that the public service can use to guide ADM use and minimise legal risk.

“[That] will give agencies greater confidence to invest in and use automated decision-making tools, particularly those that rely on artificial intelligence, while ensuring decisions remain lawful, transparent and fair,” he said.

Automation ‘frees up time’

A recent example of the rapid roll-out of ADM is the Social Security (Modernisation) Amendment Act, which was passed under urgency last month to bolster the Ministry of Social Development’s (MSD) ability to use AMD in the welfare system.

Social Development Minister Louise Upston said it will allow MSD to make millions of “straightforward” decisions via ADM, minimising unnecessary manual processing, duplication, and delay. “

“This is about delegating straightforward decision-making to an automated process, to free up time for case managers to engage better with clients so they can access assistance and move from welfare to work,” she said, noting that the ADM used by MSD does not involve generative AI.

However, Greens MP Ricardo Menendez March said Australia’s “Robodebt” fiasco showed what could go wrong when automated decision making is used without due care. The scandal involved an automated welfare debt collection system that mistakenly billed hundreds of thousands of people between 2015 and 2019 in a bid to claw back supposed overpayments.

“No matter what the government says, if you look at this bill, there’s a carte blanche expansion to basically allow a robot – a machine – to have power over people’s lives… this does remind me of Robodebt or other automated systems overseas that have been an absolute failure,” he told Parliament during debate on Upston’s bill.

“They have actually cost people’s lives. People died in Australia because of automated systems that ruined people’s lives and made mistakes, put people into debt, and it literally drove people to suicide.”

Labour’s Helen White said there were some areas, such as welfare, which were not suited to automated decision making because it risked “dehumanising” state systems.

“I’d be absolutely cautious about this because you’re talking about the very group of people who are most disconnected,” she said. “It’s very, very important that we safeguard that connection between the people who are most vulnerable and the systems we put in place, which either provide a safety net or further alienate them. That is my concern here, that we are looking at a movement that is dehumanising,”

‘Very real risks’

Automated decision making by the state does not have to involve AI, but when it does the variety of uses increases exponentially. DOC uses it to assess land-use applications, ACC uses it for real-time call centre support, NZTA has trialled it in cameras detect when motorists are using mobile phones or not wearing seatbelts.

The New Zealand Law Society (NZLS) last year identified ADM as an emerging challenge to the rule of law, saying that it can offer improved speed and efficiency but also risks lessening accountability and transparency.

“There is also a risk that ADM systems are assumed to be free from error, even where it is not possible to check how the system has reached a particular decision,” an NZLS report said.

It warned that “black box” algorithms, which prevent humans from understanding how a decision is reached, must be avoided in ADM.

“We are able to see their inputs and outputs, but we cannot see how one becomes the other… this makes it harder for individuals to understand whether decisions which impact them are in fact fair and lawful, and to then challenge those decisions.”

The Privacy Commissioner has similar concerns, telling LawNews that the use of ADM also comes with “very real risks” for the public and can affect the trust and confidence in the agencies that use it.

“These risks include inaccurate predictions, discrimination, unexplainable decisions, and a lack of accountability,” it said. “Failures in such tools are usually not evenly distributed, perpetuating or exacerbating poor outcomes for disadvantaged communities. Guardrails are necessary to ensure we can innovate safely in ways that grow and maintain trust in government decision-making.”

If in doubt, leave it out

Lloyd Gallagher, convenor of The Law Association’s technology committee, believes the Law Commission’s ADM review is premature, calling instead for legislative guardrails around the use of AI.

He said AI was a technology still in its infancy and critical legal issues, such as who should be held liable when AI causes harm or acts unlawfully, have not been resolved.

Until then, Gallagher believes the government should not use AI for decision making unless was it backed by human checks, saying unmonitored deployment of the technology is “dangerous”.

“AI should be left out of any of the government department’s [ADM] use – except as a [assistance] tool – until we have proper legislation in place to support and we don’t have enough research to create that legislation yet.”

Gallagher said it did not make sense for the Law Commission to examine how AI can be used in ADM before the legal uncertainties around AI itself had been thrashed out.

“I’m not sure why we’re jumping into implementation before we even have a legislation to protect against the harms.”

“Before we start looking at how we’re using the technology, we must first start with how we make sure society is protected against misuse of the technology.”

Hickford said government departments were already using in automated decision-making and the technology would be included in the commission’s review, at least in the initial stages.

No timeline has been given for the review’s completion.