Fix "[Errno 32] Broken pipe" in Python
One day, I’ve tried to run a python script using the multiprocessing technique and for a while the program crashed and raised the [Errno 32] Broken pipe error…
Overview
One day, I’ve tried to run a python script using multiprocessing technique n_jobs=10 and for a while the program crashed and raised the [Errno 32] Broken pipe error.
With some google searches I founded the problem as well the solution for it.
“Broken pipe” is essentially an IOError error (short for input/output error), which happened at the Linux system level. It usually occurs when reading and writing files, or in other words, doing file input/output or network input/output (via sockets) 1.
In programs that uses worker processes to speed up processing and make use of multi-core CPUs, you can try reducing the number of the worker processes to see whether the error disappear or not 1.
From that suggestion, I reduced n_jobs=10 to n_jobs=5 and boom, the error got disappeared.
