Goodhart Strikes Again. How the old statistical trap of… | by Mikey Shulman | Kensho Blog

How the old statistical trap of Goodhart’s law is more pervasive in machine learning than we realize. In the world of finance and financial data, ears will often prick up when sentiment analysis is mentioned. Put simply, knowing how the public feels about a certain topic, for example a public company, commodity, or abstract concept like ESG, presents a straightforward way to make money. However, a working sentiment analyzer for finance, to be used as a proxy for asset prices, has evaded practitioners, largely because the sentiment signals are rarely predictive of asset prices. Indeed, predicting returns is difficult, but the reason for the ubiquity of failures has a different cause, namely Goodhart’s law. Goodhart’s law, which states that every statistical regularity tends to collapse when pressure is applied to it, implies that once a metric becomes a target, it ceases being a useful metric. Machine learning practitioners are familiar with Goodhart’s law in two scenarios: first, it ha

How the old statistical trap of Goodhart’s law is more pervasive in machine learning than we realize. In the world of finance and financial data, ears will often prick up when sentiment analysis is mentioned. Put simply, knowing how the public feels about a certain topic, for example a public company, commodity, or abstract concept like ESG, presents a straightforward way to make money. However, a working sentiment analyzer for finance, to be used as a proxy for asset prices, has evaded practitioners, largely because the sentiment signals are rarely predictive of asset prices. Indeed, predicti