The problem
"Yes, you can use this bot for professional business advice," New York City's MyCity Business chatbot replied, days after the city had added a disclaimer to its page telling users the opposite. The bot was presented as reaching trusted information across the city's own business web pages. What it produced included answers that cash-free restaurants, tenant lockouts and taking workers' tips were permitted, plus one housing question answered both ways. The city shipped a disclaimer. The service came down in February 2026, on budget grounds.
Blast radius
NYC business owners seeking compliance guidance
Time to detect
About five months to external testing
Recovered
Partial — Mitigations only; no accuracy fix on record
01 — Context
New York City launched MyCity Business at chat.nyc.gov in October 2023, under Mayor Eric Adams, to help business owners navigate city regulation. Microsoft's Azure AI services power it. The city presented the tool as giving users access to "trusted information from more than 2,000 NYC Business web pages," and the bot described itself as "trained to provide you official NYC Business information." A city spokesperson called it a "pilot program" as of March 2024. What matters more than the product framing is the subject matter: these are compliance answers, and a wrong one is acted on by the person who asked it.
02 — What failed
Markup reporters queried the live bot, joined by housing expert Rosalind Black, and compared answers across more than ten staffers. Asked whether a restaurant could go cash-free, the bot answered: "Yes, you can make your restaurant cash-free. There are no regulations in New York City that require businesses to accept cash as a form of payment." In 2020 the city council had passed a law requiring businesses to accept cash, precisely to prevent discrimination against unbanked customers. The bot also said it was legal to lock out a tenant, that "there are no restrictions on the amount of rent," that it was "fine to take workers' tips," and that there are "no regulations on informing staff about scheduling changes"; it suggested it was "OK to conceal funeral service prices." But the sharper finding was not any one wrong answer. Asked whether landlords must accept housing vouchers, the bot told ten separate Markup staffers "no," and told one reporter "yes." One question, answered both ways.
03 — Symptoms
A practitioner meets this as "chatbot gives different answers to the same question," "assistant contradicts its own source documents," "AI assistant confidently wrong about regulations," or "disclaimer doesn't stop chatbot giving advice." Nothing in the output itself flagged the problem. The wrong answers were fluent, direct and formatted exactly like every correct answer, so a business owner reading one response had nothing to work with. The tell only exists in comparison: the same housing-voucher question produced ten answers of "no" and one of "yes," and cross-user comparison is not something any single user can ever perform. Nor can a user tell that the bot will affirm it is suitable for professional business advice while the page around it says the opposite.
04 — Root cause
The root cause available from the public record is organisational rather than architectural, and it is worth saying plainly why: no vendor or city engineering account of this system exists. What the record does establish is a sequence of institutional choices. A system was deployed into a regulated-advice domain — business, labour and housing law — while being represented as trained on official city business information, with Microsoft afterwards stating its goal of outputs "accurate and grounded on the city's official documentation." At production that representation did not hold, and it failed in two separable ways: answers that contradicted rules on the books, and one question that returned contradictory answers depending on who asked. Grounding was asserted and not achieved. Then, once the failures were documented, what shipped was a disclaimer and a narrowing of scope rather than a demonstrated correction — a response the bot itself contradicted days later, which is what puts the organisational handling, rather than any single wrong answer, at the centre of this case.
05 — Technical explanation
The architectural content of the record is one line: Microsoft's Azure AI services powered the chatbot. Everything else around it describes what was claimed rather than what was built — a corpus characterised as more than 2,000 NYC Business web pages, a bot that described itself as "trained to provide you official NYC Business information," and a Microsoft spokesperson's statement, given after the failures were reported, that the company would keep working "to improve the service and ensure the outputs are accurate and grounded on the city's official documentation." That last line is a forward-looking statement of intent for the system, not an account of how it was originally assembled. No primary source names the mechanism by which the bot was supposed to reach those web pages. What the observed behaviour does support is narrower, and still useful: an identical query returning contradictory answers shows that answers were not determined by a stable lookup against an authoritative source, because a stable lookup does not return "yes" and "no" to the same question. What it cannot show is which stage of the system produced the divergence. The public record does not permit that call, and this case does not make it.
06 — Contributing factors
The subject matter carried risk of its own: compliance advice, where following a wrong answer creates legal exposure for the person who followed it. The framing was also doubled. The tool was described as a "pilot program" and later relabelled a "beta product" while remaining publicly available and describing itself as trained on official city business information. Officials had publicly hailed the bot's accuracy; in undated promotional remarks on a panel, Small Business Services Commissioner Kevin D. Kim invoked the Air Canada chatbot bereavement-refund lawsuit as the thing that must not happen to a government — "That can't happen to government," he said, and "We cannot be in a situation where we lose that kind of trust even one time." The risk was understood in the abstract before it materialised. Julia Stoyanovich, Computer Science Professor and Director of the Center for Responsible AI at New York University, called the approach "reckless and irresponsible": "They're rolling out software that is unproven without oversight. It's clear they have no intention of doing what's responsible."
07 — Attempted fixes
Mayor Adams committed to a repair: "We're identifying what the problems are, we're gonna fix them, and we're going to have the best chatbot system on the globe," adding that "we took the whole story and we gave it over to the team and said, 'We've got to fix these problems.'" What shipped was text. A new disclaimer told users to "Always double-check its information using the provided links or by visiting MyCity Business. Do not use its responses as legal or professional advice or provide sensitive information to the Chatbot," and the page relabelled the tool a "beta product" that may give "inaccurate or incomplete" responses, replacing earlier wording about "incorrect, harmful or biased content." The administration also appeared to limit the kinds of questions the tool was willing to answer, in the hedged wording of the reporting. Days after the disclaimer shipped, the bot was asked whether it could be used for professional business advice and answered "Yes, you can use this bot for professional business advice." No correction to the underlying answers is on the record, and the record is likewise silent on the bot's accuracy after April 2024. The city took the chatbot down on February 4, 2026, and the actionable reason given was cost rather than accuracy: Mayor Mamdani called it "functionally unusable," while a spokesperson said the transition team "presented it to the mayor as a possible place to save funds" amid the city's budget gap. Building the bot's foundations reportedly cost nearly $600,000, and Mamdani put its ongoing cost at around half a million dollars.
08 — Lessons learned
A grounding claim is a claim. A vendor stating a goal of outputs "accurate and grounded on the city's official documentation," or a system describing itself as trained on official information, settles nothing; only an independent check against the source documents can settle it. Inconsistency on an identical question deserves separate treatment. It is a release-blocking signal rather than a known limitation to disclaim around, and because it is invisible to any individual user, somebody has to deliberately test for it. A disclaimer relocates liability without improving an answer, and in a compliance domain the person harmed is the one who followed the answer, not the institution that published it. And a mitigation is itself a change that needs verifying: this one was contradicted by the system it was meant to cover within days.
09 — Prevention checklist
- Audit answers against the underlying source documents on every regulated topic before public launch, rather than trusting a grounding claim.
- Require the system to cite the exact source passage behind any legal or regulatory answer, so a user can check it.
- Ask the same high-stakes question repeatedly and across users, and treat inconsistent answers as a release blocker.
- Never let a disclaimer stand in for a correctness fix where following the answer creates legal exposure.
- Re-test the live system after shipping any mitigation, since the mitigation is a change like any other.
References
-
News
-
News
- Mamdani to kill the NYC AI chatbot we caught telling businesses to break the law (The Markup) (opens in a new tab)
themarkup.org
News
- Incident 714: Microsoft-Powered New York City Chatbot Advises Illegal Practices (AI Incident Database) (opens in a new tab)
incidentdatabase.ai
Incident database
-
Incident database
-
News
10 — Solutions (3)
Verify grounding against source documents instead of asserting it
Root cause addressed
This addresses a system represented as grounded in the city's official documentation while producing answers that contradicted rules actually on the books.
Treat inconsistency on identical queries as a release blocker
Root cause addressed
This addresses the same housing-voucher question returning "no" to ten staffers and "yes" to one reporter, a failure no individual user was positioned to observe.
Never let a disclaimer substitute for a correctness fix
Root cause addressed
This addresses a shipped response that consisted of a disclaimer, which the system then contradicted days later by affirming that it could give professional business advice.
Fixed this yourself? Add the steps that helped with City chatbot told business owners illegal practices were legal.
Solution received
Thank you for sharing what worked. Your solution will be reviewed by a human before it appears alongside the others. It does not publish automatically.