scikit-learn n_jobs parameter on CPU usage & memory
I can imagine a value of -1 consumes all available resources as and when they become available. Depending on which function you are talking about, it seems data is copied for each of the jobs, which can lead to memory problems if the dataset is large enough. Here is a snippet of information from the docstring of GridSearchCV:


If `n_jobs` was set to a value higher than one, the data is copied for each
point in the grid (and not `n_jobs` times). This is done for efficiency
reasons if individual jobs take very little time, but may raise errors if
the dataset is large and not enough memory is available.  A workaround in
this case is to set `pre_dispatch`. Then, the memory is copied only
`pre_dispatch` many times. A reasonable value for `pre_dispatch` is `2 *
n_jobs`.



So it might be a good idea to use pre_dispatch to put an upper limit on your memory consumption.

Otherwise, why are you setting it to -1? You should just set it to the number of physical cores on your machine, or maybe 2 times that number, if the task can be multi-threaded. 

EDIT:

It seems setting n_jobs=-1 does indeed just select all physical cores and maximises their usage. Have a look at the comments in this answer on StackOverflow.

If you have not set pre_dispatch, it will of course try to copy a lot. This is why you run out of memory. If you have 4 cores there will be, by default, 8 copies of the dataset made (as described above in the quote).

Here is another thread, which looks more at the performance side