Measuring Fairness

There are multiple ways to measure accuracy. No matter how we build our model, accuracy across these measures will vary when applied to different groups of people.

Measuring Fairness Explorables Measuring Fairness There are multiple ways to measure accuracy. No matter how we build our model, accuracy across these measures will vary when applied to different groups of people. Measuring Fairness How do you make sure a model works equally well for different groups of people? It turns out that in many situations, this is harder than you might think. The problem is that there are different ways to measure the accuracy of a model, and often it's mathematically impossible for them all to be equal across groups. We'll illustrate how this happens by creating a (f

related reading

• Statistical Fairness - Turing Commonsalan-turing-institute.github.io
• Can a Model Be Differentially Private and Fair?pair.withgoogle.com
• Classificationfairmlbook.org
• Relative notions of fairnessfairmlbook.org
• Too much efficiency makes everything worse: overfitting and the strong version of Goodhart’s law | Jascha’s blogsohl-dickstein.github.io
• A statistical approach to model evaluations \ Anthropicanthropic.com
• Frontiers | On Consequentialism and Fairnessfrontiersin.org
• Audits Under Resource, Data, and Access Constraints: Scaling Laws For Less Discriminatory Alternativesarxiv.org
• Goodhart's Law and Why Measurement is Hard — Ribbonfarmribbonfarm.com
• Sensitivity and specificity - Wikipediaen.wikipedia.org
• Why don’t we screen healthy people to catch diseases early? | Out-Of-Pocketoutofpocket.health
• Preface - The Emerging Science of Machine Learning Benchmarksmlbenchmarks.org