Why multiprocessing suddenly stops after a certain number of tasks?
I'm trying to write some code to parallelize a bunch of tasks. Basically, the script is organized as the following.

import multiprocessing as mp


def obj_train(x):
    return x.train()


class ServerModel(nn.Module):
    self.S = nn.Parameter(torch.rand(x, y), requires_grad=True)


class ClientModel(nn.Module):
    self.S = nn.Parameter(torch.rand(x, y), requires_grad=True)
    self.U = nn.Parameter(torch.rand(x, y), requires_grad=True)


class Server:
    def __init__(self, model):
        self.model = model
        ...

    def train(clients):
        for i, c in enumerate(clients):
            sd = c.model.state_dict()
            sd['S'] = self.model.S
            c.model.load_state_dict(sd)
        self.c_list = random.sample(clients, 200)
        pool = mp.Pool(mp.cpu_count()-1)
        results = pool.map(obj_train, self.c_list)
        pool.close()
        pool.join()
        print("Training complete")


class Client:
    def __init__(self, client_id, model, train_set):
        self.id = client_id
        self.model = model
        self.train_set = train_set

    def train(self):
        self.optimizer = optim.SGD([self.model.S, self.model.U])
        for i in self.train_set:
            loss = self.model(i)
            loss.backward()
            self.optimizer.step()
        print("Trained client %d", self.id)

        return self.model.S


if __name__ == '__main__':
    ...
    server = Server(server_model)
    clients = [Client(u, ClientModel(), train_set[u]) for u in range(n_clients)]
    server.train(clients)


Ok, the problem is in multiprocessing. I tried with a lot of approaches but all of them gives me the same problem. Server should manage the training of 200 clients, but after a certain number of trainings (it depends on the approach, but approx 50-100), the script completely stucks and cores of the CPU stop working.

Have you any ideas? Other approaches I tried are for example mp.Pool and with ProcessPoolExecutor.

Thank you for your help.

Could it be that you hit the maximum number of processes/threads your machine is able to handle? 
It is common, for example, when moving a web crawler from development to production that the machine does not allow more processes. 

I would give a look at the file 

/etc/sysctl.d


and in case increase the number of possible processes for the machine to handle. 

Another reason might be that you capped RAM limit or something similar, try to give another quick look at the command 

htop


followed by 

free -m 


and see what they tell you. It might be a hardware problem. While from a software it might be that the library you are using https://docs.python.org/2/library/multiprocessing.html has a hard-coded limit. Also here you can easily set it higher within the library parameters. 

Last but not least, try to find the problem incrementally. I would test it with with 2 processes and increment slowly to see when the application starts having issues. And at that point it would probably be even clearer what the issue was. Good luck!
