BIG-bench/bigbench/benchmark_tasks/convinceme at main · google/BIG-bench

Beyond the Imitation Game collaborative benchmark for measuring and extrapolating the capabilities of language models - BIG-bench/bigbench/benchmark_tasks/convinceme at main · google/BIG-bench

All tasks | Interact with tasks | Reviewer discussion | 45500 multiple choice and 2275 free text queries of dummy model | Keywords: alignment , free response , game play , multiple choice , programmatic , self evaluation , truthfulness Convince Me This task probes the ability of large language models (LLMs) to produce convincing arguments for false statements, e.g., convincing someone that the moon landing was faked. Authors: Cedrick Argueta, Princeton University ( cedrick@princeton.edu ), Vinay Ramasesh, Google Research ( ramasesh@google.com ), Jaime Fernández Fisac, Princeton University ( jf

related reading

• BIG-bench/bigbench/benchmark_tasks/keywords_to_tasks.md at main · google/BIG-benchgithub.com
• Modifying LLM Beliefs with Synthetic Document Finetuningalignment.anthropic.com
• truthfulQA_lin_evans.pdfowainevans.github.io
• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
• [2503.03750] The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemsarxiv.org
• 2308.03958arxiv.org
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• [2203.02155] Training language models to follow instructions with human feedbackarxiv.org
• LMCA_dataset.pdfandrew.cmu.edu
• [2303.17548] Whose Opinions Do Language Models Reflect?arxiv.org
• [2605.21739] AttuneBench: A Conversation-Based Benchmark for LLM Emotional Intelligencearxiv.org
• Measuring what Matters: Construct Validity in Large Language Model Benchmarksalphaxiv.org