HealthBench: Evaluating Large Language Models Towards Improved Human Health

Today's paper introduces HealthBench, a comprehensive benchmark for evaluating large language models (LLMs) in healthcare contexts. HealthBench consists of 5,000 multi-turn conversations between models and users, with responses evaluated using conversation-specific rubrics created by 262 physicians. Unlike previous multiple-choice or short-answer benchmarks, HealthBench enables realistic, open-ended evaluation through 48,562 unique rubric criteria spanning various health contexts and behavioral dimensions.

Overview

HealthBench evaluates LLMs through a rubric-based approach applied to realistic health conversations. Each evaluation example consists of a conversation between a model and a user (either an individual or a healthcare professional), along with a set of rubric criteria specific to that conversation. These criteria describe attributes that a good response should include or avoid, with each criterion assigned a point value between -10 and 10. A model-based grader determines whether each criterion is met, and the final score is calculated by summing the points for met criteria and dividing by the maximum possible score.

The benchmark was developed with input from 262 physicians with practice experience across 60 countries and 26 medical specialties. These physicians helped create the conversations, categorize them into themes, and develop the rubric criteria. Most conversations were synthetically generated using a language model pipeline designed to create realistic health interactions, while others came from physician red-teaming exercises or were derived from frequently-searched health queries.

HealthBench organizes its 5,000 examples into seven themes that reflect different areas of health interactions: emergency referrals, context-seeking, global health, health data tasks, expertise-tailored communication, responding under uncertainty, and response depth. Each rubric criterion is also categorized along five axes of model behavior: accuracy, completeness, communication quality, context awareness, and instruction following. This structure allows for detailed analysis of model performance across different dimensions.

The benchmark includes two variations: HealthBench Consensus, which contains only criteria validated by multiple physicians, and HealthBench Hard, a subset of 1,000 particularly challenging examples. These variations provide additional perspectives on model performance, with Consensus offering higher precision in identifying model failures and Hard providing a challenging target for future models.

Results

HealthBench reveals significant progress in LLM performance on healthcare tasks over time. Recent models have shown substantial improvements, with OpenAI's o3 model achieving a score of 60%, compared to 16% for GPT-3.5 Turbo and 32% for GPT-4o. The benchmark also demonstrates that smaller, more cost-effective models have improved dramatically, with GPT-4.1 nano outperforming GPT-4o while being 25 times cheaper.

Performance varies across themes and axes, with models generally performing better on emergency referrals and expertise-tailored communication than on context-seeking, health data tasks, and global health. Similarly, models tend to score higher on accuracy, communication quality, and instruction following than on completeness and context awareness.

The reliability of models has also improved, with o3 achieving more than double the worst-at-16 score of GPT-4o, though there is still substantial room for improvement. When comparing model responses to physician-written responses, recent models outperformed unassisted physicians, and physicians were able to improve responses from older models (August/September 2024) but not from the newest models (April 2025).

On HealthBench Hard, the most challenging subset, even the strongest model (o3) achieves only a 32% score, indicating significant headroom for future improvements. The benchmark also shows that model-physician agreement on consensus criteria is similar to physician-physician agreement, suggesting that the evaluation is trustworthy.

Conclusion

HealthBench represents a significant advancement in evaluating LLMs for healthcare applications. By providing a comprehensive, physician-validated benchmark that measures performance across diverse health contexts and behavioral dimensions, it offers a meaningful standard for assessing and improving model safety and effectiveness. The results demonstrate substantial progress in recent models but also highlight areas where further improvement is needed, particularly in context-seeking behavior and reliability. For more information please consult the full paper.

Congrats to the authors for their work!

Arora, Rahul K., et al. "HealthBench: Evaluating Large Language Models Towards Improved Human Health." OpenAI, 2024.