Dataset Card for PARROT

Performance Assessment of Reasoning and Responses on Trivia (PARROT) is a validated LLM benchmarking dataset that leverages game show data for a more realistic evaluation of Large Language Models (LLMs). Curated by Redblock, this dataset offers unique challenges through its open-ended and closed-ended question formats, derived from the popular game shows Jeopardy and Who Wants to Be a Millionaire.

Dataset Details

Dataset Description

PARROT is designed to provide a robust evaluation of LLM performance through diverse QA tasks. It is comprised of two distinct datasets:

1. PARROT-Jeopardy: A dataset consisting of questions from the game show Jeopardy, featuring short, concise questions for testing reasoning and ambiguity handling.

2. PARROT-Millionaire: A dataset consisting of questions from the game show Who Wants to Be a Millionaire, known for its straightforward nature and broad range of topics. This dataset is valuable for evaluating an LLM's knowledge.

• Curated by: Redblock
• Shared by: Redblock
• License: cc-by-4.0

Uses

Direct Use

PARROT is designed to benchmark the performance of Large Language Models (LLMs) in Question-Answering tasks, particularly over trivia.

Dataset Structure

PARROT-Jeopardy

• ep_num: Episode number from the season.
• air_date: Date when the episode aired.
• extra_info: Additional episode information, including the host's name.
• round_name: The round being played (e.g., Jeopardy, Double Jeopardy, Final Jeopardy).
• coord: Coordinates of the clues on the game board.
• category: Clue category.
• value: Monetary value of the clue.
• daily_double: Boolean indicating if the clue is part of the Daily Double round.
• question: The clue itself.
• answer: Labeled answer or guess.
• correct_attempts: Count of contestants who answered correctly.
• wrong_attempts: Count of contestants who answered incorrectly.

PARROT-Millionaire

• question_info: Describes the price value and the current question number.
• question: The question in text form.
• options: Four predefined options corresponding to the question.
• correct_answer: Labeled correct answer.
• price: Engineered feature from Question Info, indicating the dollar value of the question.
• normalized_options: Engineered feature providing text normalization for the options.
• normalized_correct_opt: Engineered feature providing text normalization for the correct answer.

Dataset Creation

Curation Rationale

PARROT was created to address the need for a more realistic and challenging benchmarking dataset for LLMs. By using game show data, the dataset captures a wide range of question types and difficulties, providing a comprehensive evaluation tool.

Source Data

Data Collection and Processing

• Data for PARROT-Jeopardy was curated from seven key seasons of Jeopardy the game show to ensure a representative sample across the show's timeline. The data was scraped from the J!Archive, a fan-created archive containing over 500,000 clues.

• PARROT-Millionaire was created by scraping data from the Millionaire Fandom site. The data was organized and processed to ensure consistency and reliability.

Who are the source data producers?

The original data for PARROT-Jeopardy was sourced from the fan-created archive of the original show Jeopardy, while the data for PARROT-Millionaire was sourced from the Millionaire Fandom site.

Personal and Sensitive Information

The dataset does not contain personal, sensitive, or private information.

Citation

BibTeX: If you use this dataset in your research, please cite it as follows:

@dataset{parrot2024,
author = {Redblock AI Team},
title = {PARROT: Performance Assessment of Reasoning and Responses on Trivia},
year = 2024,
publisher = {Redblock},
url = {https://huggingface.co/datasets/redblock/parrot},
license = {CC BY 4.0}
}

APA:

Redblock AI Team. (2024). PARROT: Performance Assessment of Reasoning and Responses on Trivia. Redblock. Available at https://huggingface.co/datasets/redblock-ai/parrot.

More Information

For more information, visit redblock.ai.

Dataset Card Authors

Redblock AI Team

Dataset Card Contact

For inquiries, visit redblock.ai.

dataset_info.json

{
  "description": "PARROT is a validated LLM benchmarking dataset that leverages game show data for a more realistic evaluation of Large Language Models (LLMs). Curated by Redblock, this dataset offers unique challenges through its open-ended and closed-ended question formats, derived from the popular game shows Jeopardy! and Who Wants to Be a Millionaire?",
  "homepage": "https://huggingface.co/datasets/redblock-ai/parrot",
  "license": "cc-by-4.0",
  "citation": "@dataset{parrot2024,\nauthor = {Redblock AI Team},\ntitle = {PARROT: Performance Assessment of Reasoning and Responses on Trivia},\nyear = 2024,\npublisher = {Redblock},\nurl = {https://huggingface.co/datasets/redblock/parrot},\nlicense = {CC BY 4.0}\n}",
  "version": "1.0.0",
  "splits": {
    "jeopardy": {"num_examples": 61462},
    "millionaire": {"num_examples": 22698}
  },
  "download_size": 20000000,
  "dataset_size": 20000000
}

Disclaimer

Important Notice:
The datasets curated in this benchmark include content derived from fan-created sites related to Who Wants to Be a Millionaire? Fandom and J! Archive. These datasets are intended solely for research, educational purposes, and non-commercial use. Redblock does not claim ownership of, nor does it have any affiliation with, the creators or copyright holders of Who Wants to Be a Millionaire? and J! Archive.

Who Wants to Be a Millionaire? Fandom and J! Archive are registered trademarks of their respective owners. Redblock's use of these materials is protected under the fair use doctrine as defined by U.S. copyright law, which permits the use of copyrighted material for purposes such as criticism, commentary, news reporting, teaching, scholarship, and research.

Redblock has modified these datasets in compliance with U.S. law to ensure that the content remains within the boundaries of fair use. Any modifications or derived works created from these datasets should also adhere to the principles of fair use and respect the intellectual property rights of the original content creators.

This benchmark is provided by Redblock "as-is" without any guarantee of accuracy or fitness for a particular purpose. Users of this benchmark are encouraged to respect copyright laws and the intellectual property rights of the original content creators. The datasets should not be used for commercial purposes without obtaining proper authorization from the rights holders.