WebGPT Comparisons

WebGPT Comparisons contains all comparisons that OpenAI marked as suitable for reward modeling by the end of the WebGPT project. Each record pairs two model-generated answers to the same question, and each answer is accompanied by the quotes (extracts plus page titles) that the model retrieved while browsing. Human labelers assigned every answer a preference score from -1 to 1 (the two scores in a pair sum to 0), so the answer with the positive score is the preferred one; ties are scored 0 and treated as soft 50% labels while all other scores are used as hard labels via their sign. The dataset also stores GPT-2 token sequences (a prefix built from the question and quotes, plus the answer completion) whose concatenation was the exact input used for reward modeling. There are 19,578 comparisons in total, and each question is tagged with the source dataset it was taken from (e.g. triviaqa in the released rows) and a unique id. The data underpins the reward model described in the WebGPT paper (arXiv:2112.09332).