Telling Your Chatbot You Have a Mental Health Condition Can Change the Answer You Get

New research indicates that disclosing a mental health condition to AI agents can significantly alter their responses, even for benign tasks. A preprint study led by Northeastern University researcher Caglar Yildirim found that large language models (LLMs) become more likely to refuse requests, including legitimate ones, after such disclosures.

Study Methodology and Findings

The study utilized a benchmark called AgentHarm, testing AI models under three conditions: no background information, a short user bio, and the same bio with an added line stating the user has a mental health condition. Researchers observed that across tested models, including DeepSeek 3.2, GPT 5.2, Gemini 3 Flash, Haiku 4.5, Opus 4.5, and Sonnet 4.5, the inclusion of mental health context led to a decrease in the completion of harmful tasks. However, this also resulted in a higher rate of rejection for legitimate requests.

“The takeaway is that even a small cue can shift model behavior,” Yildirim stated, emphasizing that the effect might not be uniform across all phrasing. The research suggests a potential specificity to the mental health disclosure cue, though systematic variations in phrasing were not explored.

Context and Implications

This research emerges as AI agents become more prevalent and developers integrate memory features for personalized user experiences. It also follows a period of increased scrutiny on AI developers due to alleged links between AI interactions and real-world harm, including suicides and violent crimes. OpenAI reported over 1 million weekly discussions about suicide with ChatGPT, and a lawsuit has been filed against Google claiming Gemini contributed to a user's suicide.

The study highlights that the observed effect varies by model and can be influenced by external factors. When LLMs were subjected to “jailbreak” prompts designed to bypass safety measures, the impact of the mental health disclosure weakened or broke down. “A model might look safe in a standard setting, but become much more vulnerable when you introduce things like jailbreak-style prompts,” Yildirim explained. He noted that in agent systems, which plan and act over multiple steps, a combination of strong instruction-following capabilities and easily bypassed safeguards can increase risk.

Potential Explanations and Limitations

While the study does not pinpoint a single cause for the behavioral shifts, it suggests possibilities such as safety systems reacting to perceived user vulnerability, keyword-triggered filtering, or altered prompt interpretations due to personal details. Yildirim also cautioned that the findings are based on AI reviewer scores and may not definitively reflect real-world harm. He added that stylistic changes in AI output, such as refusal-adjacent responses, could also be influenced by personalization conditions.