Skip to main content
Artificial Intelligence

A clinically validated framework for auditing AI chatbot behavior in mental health interactions.

| Source: Nature medicine

Millions of users turn to consumer artificial intelligence chatbots to discuss emotional, behavioral and mental-health concerns, creating an urgent need for rigorous and scalable safety evaluations. Here we introduce simulated (SIM) vulnerability-amplifying interaction loops (VAILs) (SIM-VAIL), a clinically validated framework for auditing chatbot behavior in mental-health contexts. SIM-VAIL simulates users with specific psychiatric vulnerabilities and conversational intents, engages them in mul

Millions of users turn to consumer artificial intelligence chatbots to discuss emotional, behavioral and mental-health concerns, creating an urgent need for rigorous and scalable safety evaluations. Here we introduce simulated (SIM) vulnerability-amplifying interaction loops (VAILs) (SIM-VAIL), a clinically validated framework for auditing chatbot behavior in mental-health contexts. SIM-VAIL simulates users with specific psychiatric vulnerabilities and conversational intents, engages them in multi-turn conversations with frontier artificial intelligence chatbots (including Claude, ChatGPT, Gemini, Grok and Llama models) and scores each exchange across 13 clinically grounded risk dimensions. Across 810 conversations, spanning 9 target chatbots and 30 simulated user profiles, concerning behavior in target chatbots was widespread, albeit reduced in newer models. Concerning behavior varied by user vulnerability and conversational intent, accumulated over turns, and could be reduced by interventions at early escalation points. Risk was highest when otherwise supportive chatbot behaviors reinforced the psychological mechanisms underlying the simulated user's vulnerability, a pattern we term a VAIL. SIM-VAIL provides a scalable framework for mapping mental-health risk across users, chatbots and conversational trajectories, offering a foundation for targeted safety improvements.

Read the original source →

Related Stories

Artificial Intelligence

Where intention goes, responsibility flows: Grounding AI accountability in the machine learning paradigm.

An opening salvo of lawsuits in US courts has begun holding AI companies accountable through independent doctrinal routes: Product liability and agency law cut as swords, while the shields of Section 230 immunity and First Amendment doctrine begin to crack. This Perspective points out that these converging paths share a unifying principle, built on the very engineering architecture of machine learning (ML) itself. Every ML system, across all variants, is constituted by a human-specified objectiv

Continue reading
Artificial Intelligence

Intermittent gas-liquid rearrangement controls the mixing of gases in porous media.

Gas mixing in porous media governs important environmental and industrial applications in which multiple gas components share the available pore space with liquid water, simultaneously undergoing gas mixing and gas-liquid redistribution. Currently, this process is represented by macroscopic models that ignore pore scale dynamics. We demonstrate that this is not generally valid, and can lead to large errors in the mixing rate. We use 4D X-ray imaging to investigate nitrogen injection into a porou

Continue reading
Artificial Intelligence

Normalized Raman imaging for studies of tissue physiology of the kidney.

Conventional histology relies on fixation, embedding, sectioning, and staining methods that distort cellular architecture, extract lipids, and introduce variability, limiting reproducibility. We present a pipeline for automated histopathology in the mouse kidney using normalized stimulated Raman imaging (NoRI) that enables quantitative, label-free protein and lipid measurements at high spatial resolution. NoRI computationally corrects protein, lipid, and water Raman signals, allowing quantitativ

Continue reading
Artificial Intelligence

Opportunistically Detecting Signs of Hypertension on a Consumer Smartwatch.

Hypertension is a silent killer, with over half of affected adults unaware of their condition 1,2 . This lack of awareness is a major concern, as early intervention is critical for preventing major adverse cardiovascular events 3,4 . Here we show that applying artificial intelligence (AI) to multimodal waveforms (photoplethysmography and accelerometry) recorded using a widely available consumer smartwatch (AI-PPG-ACC-HTN), without any cuff calibration, can detect hypertension with accuracy level

Continue reading
Artificial Intelligence

Eating disorders.

The eating and feeding disorders are serious psychiatric disorders at the mind-body interface, often with an exceptionally high burden for affected individuals, their families and supporters, and wider society. In anorexia nervosa, high acute medical risk and patients' ambivalence about treatment can make a challenging combination that can impede engagement. First-line treatments for eating disorders are mainly psychological, with medications playing an adjunctive role. Novel biological and tech

Continue reading