Python multiprocessing in 5 minutes
I was missing an easy-to-read example of the basic use of Python Multiprocessing so I decided to write one.
Conceptual review: we want multiprocessing because we have access to machines with multiple CPU cores (either virtual or physical) and without multiprocessing all potentially parallelizable work will run sequentially. Assuming we have N real cores, we would be able to divide the execution time by N. If we have N virtual CPUs each of which mapped to a physical core with SMT, the factor would be (N/2 + delta), where delta is a number considerably lower than N/2, probably between 0.1 * N/2 and 0.2 * N/2. Whatever the case is, there is a very large potential reduction of execution time for parallelizable workloads. For an 8 vcCPU virtual machine, dividing execution time by 4.4 would be a reasonable estimation of how multiprocessing is helpful. Finally, we want multiprocessing rather than threads because Python threads are not efficient due to the GIL.
This example covers the following situations:
- execution of several different functions in parallel each of which receiving different arguments
- execution of several instances of the same function in parallel each of which receiving different arguments
Here is some (hopefully) self-explanatory code:
The code above goes as far as to launch the 5 processes in background and wait for them to complete. But what happens next? Ideally, we would like to be aware of the exit code of every process before we go forward with the execution of the main program:
Obviously, for this to work we need a worker_function. Here is a fictional function that can be used with the code structure above:
Does it work, though? If you'd like to see this example in action you can simply execute:
git clone https://github.com/ghomem/python-blueprints.git
python3 python-blueprints/multiprocessing/multiproc-blueprint.py
The result should be something like this:
That's it. Hopefully, this 5 minute read saves more than 5 minutes to whoever reads it.