Python - Running multiple python scripts that uses multiprocessing affects performance and errors out sometimes
If you are running multiple scripts that use multiprocessing and write to the same disk at the same time, you will eventually hit a bottleneck somewhere.
It could be concurrent access to the database, writing speed of the disk, amount of memory used or CPU cycles. What specifically is the problem here is impossible to say without doing measurments.
But e.g. writing things to a HDD is very slow compared to current CPU speeds.
Also, when you are running multiple scripts that use multiprocessing you could have more worker processes than the CPU has cores. In which case there will be some worker processes waiting for CPU time all the time.
I have a PYTHON script that uses multiprocessing to extract the data from DB2/Oracle database to CSV and ingest to Snowflake. When I run this script, performance is good (extracts the source table that is large dataset in 75 seconds).  So I made a copy of this python script and changed the input parameters (basically different source tables). When I run all these python scripts together, performance gets an impact (for the same table, it extracts in 100 seconds) and sometimes i see an error 'Cannot allocate memory'.
I am using Jupyter Nootebook and all these different python scripts extracts different source tables to CSV files and saves it in same server location.
I am also checking on my own. But any help will be appreciated.
Thanks
Bala