flâneur

Running Python Parallel Applications with Sub Interpreters

tonybaloney.github.io · 3,430 words · saved by 1 readers

Python 3.12 introduced a new API for “sub interpreters”, which are a different parallel execution model for Python that provide a nice compromise between the true parallelism of multiprocessing, but with a much faster startup time. In this post, I’ll explain what a sub interpreter is, why it’s important for parallel code execution in Python and how it compares with other approaches. Python’s system architecture is roughly made up of three parts: To learn more about this, you should read the “Parallelism and Concurrency” chapter of my book CPython Internals. Since Python 1.5, there has been a C-API to have multiple interpreters, but this functionality was severely limited by the GIL and didn’t really enable true parallelism. As a consequence, the most commonly used technique for running code in parallel (without third party libraries) is to use the multiprocessing module. In 2017, CPython core developers proposed to change the structure of interpreters so that the they were better isola

Running Python Parallel Applications with Sub Interpreters Python 3.12 introduced a new API for "sub interpreters", which are a different parallel execution model for Python that provide a nice compromise between the true parallelism of multiprocessing, but with a much faster startup time. In this post, I'll explain what a sub interpreter is, why it's important for parallel code execution in Python and how it compares with other approaches. What is a sub interpreter? ¶ Python's system architecture is roughly made up of three parts: A Python process, which contains one or more interpreters

related reading