✳flâneur — a map of the web's best reading

Copying data is wasteful, mutating data is dangerous

pythonspeed.com · 1,004 words · saved by 1 readers

You have a large chunk of data—a NumPy array, or a Pandas DataFrame—and you need to do a series of operations on it. By default both libraries make copies of the data, which means you’re using even more RAM. Both libraries do have APIs for modifying data in-place, but that can lead to other problems, including subtle bugs. So what can you do? In this article you’ll learn to recognize and apply the “hidden mutability” pattern, which offers a compromise between the two: the safe operation of copy-based APIs, with a somewhat reduced memory usage. Considering the following function: If you call that function with an array whose values range from 30 to 60, 30 will become 0.0, 45 will become 0.5, and 60 will become 1. How much memory does this function use? If the array uses A bytes, the function will use 3*A bytes of RAM: So how can we reduce memory usage? To reduce memory usage, you can use in-place operations like += to do those operations on the original array: Similarly: In all of these

Copying data is wasteful, mutating data is dangerous Copying data is wasteful, mutating data is dangerous by Itamar Turner-Trauring Last updated 12 Jan 2023, originally created 03 Jan 2020 You have a large chunk of data—a NumPy array, or a Pandas DataFrame—and you need to do a series of operations on it. By default both libraries make copies of the data, which means you’re using even more RAM. Both libraries do have APIs for modifying data in-place, but that can lead to other problems, including subtle bugs. So what can you do? In this article you’ll learn to recognize and apply the “hidden mu

Explore this link on the map →

related reading