The Markup, now part of CalMatters, uses investigative reporting, data analysis, and software engineering to challenge technology to serve the public good. Register for Horna newsletter that delivers our stories and tools straight to your inbox.
This article is co-reported with THE CITYa nonprofit newsroom serving the people of New York. Subscribe to his newsletter, The scoop.
In a press conference this week Regarding New York City’s $12 billion budget deficit, Mayor Zohran Mamdani highlighted the previous administration’s artificial intelligence chatbot as one of “a number of different things that we’re going to pursue to save money.”
The chatbot, which was launched by the Eric Adams administration in fall 2023, aimed to provide business owners with an accessible way to check city rules and regulations. But as The Markup and THE CITY first documented, the bot provided responses that, if followed, would lead to illegal behavior by companies, like taking a cut of employee tips.
The mayor’s spokesperson, Dora Pekec, confirmed in a text message that the new administration is considering removing the chatbot. She said a member of Mamdani’s transition team saw a news report about the robot The markup And THE CITY and presented it to the mayor as a possible place to save funds.
Artificial intelligence
New York’s AI chatbot asks businesses to break the law
The Microsoft-powered bot claims bosses can accept tips from workers and owners can discriminate based on source of income.
During the press conference, Mamdani blamed Adams for the budget deficit, saying he had been handed “a poisoned chalice”. To close the deficit, he said he would raise taxes on the wealthy and corporations and look “under the hood” of the city budget for potential savings.
When reporters asked him what he might delete, he pointed to the chatbot.
“The previous administration had an AI chatbot that was functionally unusable,” Mamdani said. “It cost the administration about half a million dollars. That, in and of itself, is not something that can close that kind of gap, but it is an indication of how the money was spent by refusing to account for the true costs of these programs.”
The robot, built using Microsoft’s cloud computing platform, was part of an ambitious overhaul of digital services in New York called MyCity. The project aimed to streamline access to government, but was criticized for relying on on external contractors.
It is not clear how much it costs to maintain the chatbot. I’m just building the foundation of the bot would have cost nearly $600,000close to the figure provided by Mamdani. Pekec said they don’t yet have a date to remove the robot.
Tests by The Markup and THE CITY in 2024 showed that, despite promises from the Adams administration, the chatbot would confidently deliver incorrect and potentially harmful information to visitors, even on high-stakes topics.
When asked about housing policy, for example, the bot suggested that landlords might discriminate against tenants with Section 8 vouchers. Although it was a resource for business owners, the bot did not know the minimum wage and told users that it was acceptable to refuse to accept cash payments despite a city law to the contrary, enacted in 2020.
After The Markup and THE CITY’s initial report was published, readers continued to pepper the bot with sometimes outlandish questions, which it continued to fail to answer correctly. The Adams administration defended the botsaying it would improve with time.
“We identify the problems, we are going to solve them and we will have the best chatbot system in the world,” Adams said during a press conference. “People are going to come see what we’re doing in New York.”
City administrators quickly added disclaimers to the bot advising users “not to use its responses as legal or professional advice.” They also improved some of the bot’s responses, but also seemed to limit the type of questions the tool was willing to answer.
Today, the bot advises visitors to “only ask New York government a question” and warns that “answers may sometimes produce inaccurate or incomplete content.” Visitors must accept the bot’s limitations before using it.