Assessing Political Bias in Language Models

Researchers develop a new tool to measure how well popular large language models align with public opinion to evaluate bias in chatbots.

DALL-E Researchers develop a new tool to measure how well popular large language models align with public opinion to evaluate bias in chatbots. The language models behind ChatGPT and other generative AI are trained on written words that have been culled from libraries, scraped from websites and social media, and pulled from news reports and speech transcripts from across the world. There are 250 billion such words behind GPT-3.5, the model fueling ChatGPT, for instance, and GPT-4 is now here. Now new research from Stanford University has quantified exactly how well (or, actually, how poorly) t

related reading

• [2303.17548] Whose Opinions Do Language Models Reflect?arxiv.org
• [2306.13000] Apolitical Intelligence? Auditing Delphi's responses on controversial political issues in the USarxiv.org
• gpt-4.pdfcdn.openai.com
• Cognitive Biases in Large Language Models — LessWronglesswrong.com
• Emotion concepts and their function in a large language model \ Anthropicanthropic.com
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• [2203.02155] Training language models to follow instructions with human feedbackarxiv.org
• Automated Alignment Researchers: Using large language models to scale scalable oversight \ Anthropicanthropic.com
• Externalized reasoning oversight: a research direction for language model alignment — AI Alignment Forumalignmentforum.org
• Projects | Karina Nguyenkarinanguyen.com
• [2407.12856] AI-AI Bias: large language models favor communications generated by large language modelsarxiv.org
• State of AI 2025: 100T Token LLM Usage Study | OpenRouteropenrouter.ai