flâneur

Analyzing Data 180,000x Faster with Rust

willcrichton.net · 5,370 words · saved by 1 readers

This note documents one of my recent adventures in performance optimization with Rust. By following along, hopefully you’ll learn something about how to write fast Rust. Here’s the context: imagine you have data from an online exam where a set of users answered a set of questions. The raw data looks like this: Note that each user only answered a subset of all possible questions, and all scores are either 0 or 1. Here’s the problem: given a size k k, which set of k k questions has the highest correlation with overall performance? We’ll call this the k-CorrSet problem. A simple brute-force algorithm for solving the k-CorrSet problem looks like this pseudocode: We are going to implement several variations on this algorithm to see how fast we can make it. When I do data analysis, I usually start with Python and then transition to Rust when I need better speed or memory consumption. So as a baseline, let’s look at a straightforward Pandas program for solving k-CorrSet: This uses a bit of M

Analyzing Data 180,000x Faster with Rust This note documents one of my recent adventures in performance optimization with Rust. By following along, hopefully you’ll learn something about how to write fast Rust. Here’s the context: imagine you have data from an online exam where a set of users answered a set of questions. The raw data looks like this: [ { "user" : "5ea2c2e3-4dc8-4a5a-93ec-18d3d9197374" , "question" : "7d42b17d-77ff-4e0a-9a4d-354ddd7bbc57" , "score" : 1 } , { "user" : "b7746016-fdbf-4f8a-9f84-05fde7b9c07a" , "question" : "7d42b17d-77ff-4e0a-9a4d-354ddd7bbc57" , "score" : 0 } , /

related reading