AI Models Fail to Help in Mental Health Crises Despite Recognizing Danger

Scale AI has introduced DistressBench, a clinician-authored benchmark that tests whether AI models provide appropriate help during suicide or self-harm crises. Evaluating 25 frontier models across 718 clinician-written conversations, the findings reveal a gap between recognizing danger and responding appropriately: in over a third of cases where models detected a crisis, they still failed to offer help or resources; on average, they omitted nearly 30% of required response steps. Notably, when shown their own failed response alongside a better one, most models could identify the better response yet still declined to provide it. The authors hope such benchmarks give researchers concrete evidence of where responses fall short and what needs to improve, emphasizing the need for safety testing that keeps pace with how people increasingly turn to AI in their hardest moments.

(Source：LinkedIn)