title: OpenCL buffer map/unmap and enqueueWrite bandwidth
body: <p>I have a performance problem for my opencl C++ wrapper, I need to transfer data from buffer d to buffer b as fast as possible (using map/unmap to achieve 6GB/s DMA speed of pci-e) then copy that data to buffer a (at device speed about 40GB/s)</p>
  
  <pre><code>*********************************
  *       device(discrete gpu)    *
  *                               *
  *     (enqueueCopyBuffer)       *
  *   a &lt;----&gt;b                   *
  *********************************
              ^
              |(map/unmap)
              |
  ***************
  *   d------&gt;c *
  *  (memcpy)   *
  *             *
  *    host     *
  *             *
  ***************
  </code></pre>
  
  <p>I tried many combinations of ALLOC_HOST_PTR, COPY_HOST_PTR,... for a,b and c but couldnt find an optimal solution.</p>
  
  <p>Some of what I tried:</p>
  
  <pre><code> d----&gt;c (memcpy 10GB/s)
   c-----&gt;b(map/unmap CL_ALLOC_HOST_PTR)(6GB/s) ,  b----&gt;a (enqueueCopyBuffer ~5 GB/s) 
                                                (I think  ALLOC makes b host buffer)
  
   d----&gt;c (memcpy 10GB/s)
   c------&gt;b(map/unmap CL_READ_WRITE)(1.4GB/s) , b----&gt;a (enqueueCopyBuffer 40GB/s)
                       (now b is device buffer but map/unmap is not good and buggy)
  
   d------&gt;a(enqueueWriteBuf CL_READ_WRITE)(1.7GB/s) 
            (I started the new project with this)
            (multithreaded read/write does not go beyond 2GB/s)
  </code></pre>
  
  <p>but I need:</p>
  
  <pre><code> d-----&gt;c(memcpy 10GB/s)
   c-----&gt;b(map/unmap CL_???_PTR)(6GB/s) ,  b----&gt;a (enqueueCopyBuffer 40 GB/s)
  </code></pre>
  
  <p>The reason to separate a and b is, kernel execution must use device memory.</p>
  
  <p>The reason to separate d and c is, Im implementing GPU acceleration to an opensource project and I dont want to change project's array integrity.</p>
  
  <p>There are at least a dozen of a,b,c,d that I must use. 2 for velocities, 1 for pressure, ...</p>
  
  <p><strong>Question:</strong> Which buffer structure must I implement to reach the goal of "bottlenecking part must not be less than 6GB/s anywhere". Should I bundle all b's together (same for c) into a bigger buffer to do a single read/write/map/unmap for all of them?</p>