SKU RQA-5008 · Sold by External
BIG-Bench Hard (BBH)
Product specifications
| SKU | RQA-5008 |
|---|---|
| Data type | Reasoning QA |
| Volume | 6,511 examples (test split; across 27 task configs / 23 BBH tasks) |
| Size on disk | ~2.4 MB (uncompressed; 2,500,050 bytes across 27 configs per HF card) |
| Format | parquet (Hugging Face); per-task JSON in the source GitHub repo |
| Access model | PUBLIC LICENSE |
| Pricing | Free · open dataset (HF) |
| Quality score | — |
| License | MIT License |
BIG-Bench Hard (BBH) is a curated suite of 23 tasks (27 sub-task configs on Hugging Face) drawn from BIG-Bench on which prior language models failed to exceed the average human-rater score. It contains 6,511 evaluation examples, all in a single test split, spanning algorithmic, logical, commonsense, and reasoning tasks (e.g. boolean_expressions, causal_judgement, date_understanding, logical_deduction, tracking_shuffled_objects, word_sorting). Each record has just two fields: an `input` (the question, including an Options list for multiple-choice tasks) and a `target` (the gold answer). The dataset was introduced to study whether chain-of-thought (CoT) prompting closes the human-model gap; models are typically evaluated with few-shot CoT and scored by exact match on the extracted final answer. The source dataset is released under the MIT License by Suzgun et al.; the lukaemon/bbh Hugging Face mirror redistributes it as parquet.