Anthropic HH-RLHF

HH-RLHF is a collection of human preference comparisons released by Anthropic to accompany the paper 'Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback' (arXiv:2204.05862). Each preference record is a pair of multi-turn Human/Assistant dialogue transcripts that share the same context and differ only in the final assistant response: one is labelled 'chosen' (the response a crowdworker preferred) and one 'rejected'. The helpfulness data was collected in three tranches (from base context-distilled 52B models, via best-of-16 rejection sampling against an early preference model, and from an iterated online process), while the harmlessness data was collected only from base models; the Hugging Face 'default' config merges all of these into 160,800 train and 8,552 test pairs. The repository additionally ships a separate 'red-team-attempts' subset (from arXiv:2209.07858) of adversarial transcripts with harmlessness scores and attack metadata. The data is intended for research that makes models less harmful, and Anthropic warns it contains offensive and upsetting content. It is released under the MIT License.