Pytorch multiprocessing with shared memory causes matmul to be 30x slower (with only two processes)
I am trying to improve the speed of my reinforcement learning algorithm by using multiprocessing to have multiple workers generating experience at the same time.  Each process just runs the forward pass of my neural net, no gradient computation is needed.
As I understand it, when passing Tensors and nn.Modules across process boundaries (using torch.multiprocessing.Queue or torch.multiprocessing.Pool), the tensor data is moved to shared memory, which shouldn't be any slower than non-shared memory.
However, when I run my multiprocess code with 2 processes (on an 8 core machine), I find that my pytorch operations become more than 30x slower, more than counteracting the speedup from running two processes simultaneously.
I profiled my application to find which operations specifically are slowing down.  I found that much of my time was spend in nn.functional.linear(), specifically on this line inside a Tensor.matmul call:
    output = input.matmul(weight.t())

I added a timer just to this specific matmul call, and I found that when one process is running, this operation takes less than 0.3 milliseconds, but when two processes are running, it takes more than 10 milliseconds. Note that in both cases the weight matrix has been put in shared memory and passed across process boundaries to a worker process, the only difference is that in the second case there are two worker processes instead of one.
For reference, the shapes of input and weight tensors are torch.Size([1, 24, 180]) and torch.Size([31, 180]), respectively.
What could be causing this drastic slowdown?  is there some subtlety to using torch multiprocessing or shared memory that is not mentioned in any of the documentation?  I feel like there must be some hidden lock that is causing contention here, because this drastic slowdown makes no sense to me.

It seems like this was caused by a bad interaction of OpenMP (used by pytorch by default) and multiprocessing.  This is a known issue in pytorch (https://github.com/pytorch/pytorch/issues/17199) and I was even hitting deadlocks in certain configurations I used to debug.  Turning off OpenMP using torch.set_num_threads(1) fixed the issue for me, and immediately sped up my tensor operations in the multiple processes case, presumably, by bypassing internal locking OpenMP was doing.
