State attorneys general from across the United States are calling on major AI companies to address concerning mental health issues associated with AI chatbot outputs. In a recent letter, these attorneys general expressed their alarm over what they termed “delusional outputs,” urging industry leaders to take immediate action to avoid potential violations of state laws. Notable companies such as Microsoft, OpenAI, Google, and several others, including Anthropic, Apple, and Meta, have been asked to adopt a suite of internal safeguards aimed at user protection.
The initiative arises amidst a growing debate over the regulation of artificial intelligence at both state and federal levels. Key recommendations in the letter include implementing transparent third-party audits of large language models to identify signs of harmful ideations, as well as establishing incident reporting systems to inform users when chatbots produce harmful responses. The attorneys general are advocating for independent evaluations carried out by academic and civil society groups without the risk of retaliation from companies.
The letter emphasizes the dual nature of generative AI (GenAI), indicating its potential for positive societal impact while also highlighting risks, especially for vulnerable individuals. Citing incidents involving self-harm and violence reportedly linked to AI misuse, the letter underscores the urgency of addressing delusional content generated by AI systems.
To further safeguard users, the attorneys general recommend that tech firms adopt the same transparency practices for mental health incidents as they currently do for cybersecurity breaches. This includes developing clear protocols for notifying users about exposure to harmful outputs.
Additionally, the letter calls for the creation of “reasonable and appropriate safety tests” for GenAI models to ensure they do not produce potentially dangerous outputs prior to their public release.
As the conversation around AI governance heats up, federal support for AI development stands in contrast to these state-level concerns. The federal government has shown a pro-AI stance, but recent press statements from former President Trump indicate that efforts to limit state regulations on AI may soon be forthcoming.
