Scikit Learn) is n_jobs=-1 still faster than n_jobs= c when we are running multiple Random Forest in Paralllel?
The fastest in my opinion would be to use n_jobs=-1 in all 8 terminals and your computer will internally allocate as needed the necessary CPU resources to each worker.
I am currently running 8 Random Forest models(each with grid search) on 8 different datasets for 1000 times. To save time, I opened up 8 different terminals and am running each models in parallel. Now, my question is
Is it faster to train each random forest models with the argument n_jobs ==-1 or is it better to assign number of cores such as n_jobs=3? I have 26 cores available. Thus I may be able to assign at most 3 cores to each model.
Looking at the cpu usage with htop, I can see that running with n_jobs=-1 already makes full use of all cpus. Would it be possible that setting the n_jobs=-1 actually results in bottleneck when distributing the data to each core?