Assessing Political Bias in Language Models

Researchers develop a new tool to measure how well popular large language models align with public opinion to evaluate bias in chatbots.

DALL-E Researchers develop a new tool to measure how well popular large language models align with public opinion to evaluate bias in chatbots. The language models behind ChatGPT and other generative AI are trained on written words that have been culled from libraries, scraped from websites and social media, and pulled from news reports and speech transcripts from across the world. There are 250 billion such words behind GPT-3.5, the model fueling ChatGPT, for instance, and GPT-4 is now here. Now new research from Stanford University has quantified exactly how well (or, actually, how poorly) t

related reading

• [2303.17548] Whose Opinions Do Language Models Reflect?arxiv.org
• [2306.13000] Apolitical Intelligence? Auditing Delphi's responses on controversial political issues in the USarxiv.org
• 2308.03958arxiv.org
• [2203.02155] Training language models to follow instructions with human feedbackarxiv.org
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• Does ChatGPT have a liberal bias?normaltech.ai
• [2607.14345] Value Leakage: An LLM's Answers Are Silently Shaped by Its Own Valuesarxiv.org
• Cognitive Biases in Large Language Models — LessWronglesswrong.com
• Large language models are proficient in solving and creating emotional intelligence testsnature.com
• Training language models to follow instructions with human feedback.pdfproceedings.neurips.cc
• Can public chat data predict real-world AI misalignments?alignment.openai.com
• [2302.03494] A Categorical Archive of ChatGPT Failuresarxiv.org