I would like to ask for an advice, when using OpenMPI and CUDA on GPU cluster.

I am a beginner and I feel I can't foresee consequences of my decisions about a software architecture. I would highly appreciate someone's advice/rule of thumb, as the information on GPU Clusters is quite sparse.

---

Framework:
----------

**Cluster architecture**

* the cluster has 1 front node and 9 computation nodes
* the cluster is heterogeneous, every node has the Intel Xeon CPU(s) and Nvidia Tesla K80, but with different number of processors and different number of GPU cards
* the cluster runs PBSPro scheduler

**Goal**

1) redistribute the data from a root\_MPI\_process to MPI\_processes

2) load the data to GPU, execute kernel (SIMT-parallel calculations), get the results back

3) send the results back to root\_MPI\_process

4) root\_MPI\_process processes the results, creates new data
... iterate -> redistribute the data ...

*The steps 1, 2, 3 are purely* `[SERIAL]`, *and each spawned MPI\_process independent from all the others, i.e. no pieces of data are moved between any two MPI\_processes*

**My considerations for software architecture**

Alt. 1) 1 MPI process == 1 GPU

* start **`X`** MPI\_processes, every MPI\_process (except the root\_MPI\_process) is responsible for 1 GPU
* the MPI\_process then receives a chunk of data, suitable right-away to be passed to GPU and executes kernel ... steps described above

Alt. 2) 1 MPI process == 1 computational cluster node (with multiple GPUs)

* start **`X`** MPI processes, every MPI\_process (except the root\_MPI\_process) runs on 1 computational cluster node
* the MPI\_process then identifies number of GPUs, and asks for appropriate amount of data from root\_MPI\_process
* the data, passed from root\_MPI\_process to the MPI\_process, are then redistributed among available GPUs ... step 2, 3, 4 mentioned above

---

**Questions**
-------------

1) From experienced point of view, what else -- except the data passing (which is easier in 1) and more complicated in 2), from my point-of-view) -- should I consider ?

2) This application cannot take the advantage of CUDA aware MPI, because the data are not passed between GPUs, is that right ? ( Is CUDA aware MPI useful for something else then inter-GPU communication ? )

3) Solution 2) offers Universal Addressing Space with Single Address Space, but the solution 1) does not, because every MPI\_process access 1 GPU, is that right ?

---

**Edit**

* this is research in progress, and I don't dare to estimate E2E timing. For reference, this task takes approx. **60 hours on 3x GTX 1070**, the cluster has 16x Tesla K80. My computational time at the moment is quite unlimited.
* The data are approx **1 [kB] per thread**, therefore 1 kernel requires **`blocks * threads * 1024 [B]`** of data, I would like to run 1 kernel per GPU at a time.
* the kernel (each thread in each block) runs simulation of 2nd order dynamic system with evaluation of small neural network (**30 neurons**) (the number of multiplications and additions are in **100's per iteration**), there are around **1,000,000 simulation iterations** before delivering the result.

From the above I can say with confidence, that evaluation of the kernel is more time consuming than the data transfer from host<->device.