title: Can I use Quadro K4000 and K2000 for GPUDirect v2 Peer-to-peer (P2P) communictation?
body: <p>I use:</p>
  
  <ul>
  <li><strong>Single CPU</strong> <a href="http://ark.intel.com/ru/products/77781/Intel-Core-i7-4820K-Processor-10M-Cache-up-to-3_90-GHz" rel="nofollow">(Intel Core i7-4820K Ivy Bridge-E)</a> 40 Lanes of PCIe 3.0 + MotherBoard MSI X79A-GD65 (8D)</li>
  <li>WindowsServer 2012, MSVS 2012 + CUDA 5.5 and compiled as <strong>64-bit application</strong></li>
  <li>GPUs nVidia <strong>Quadro K4000 and K2000</strong></li>
  <li>All Quadros in <strong>TCC-mode</strong> (Tesla Compute Cluster)</li>
  <li>nVidia Video Driver 332.50</li>
  </ul>
  
  <p>simpleP2P-test shown that, all Quadros K4000 and K4000 - IS  capable of Peer-to-Peer (P2P), but Peer-to-Peer (P2P) access - Quadro K4000 (GPU0) &lt;-> Quadro K2000 (GPU1) : No.</p>
  
  <p>Can I use Quadro K4000 and K2000 for <a href="https://developer.nvidia.com/gpudirect" rel="nofollow">GPUDirect v2 Peer-to-peer (P2P) communication</a>?</p>
  
  <blockquote>
    <p>[C:\ProgramData\NVIDIA Corporation\CUDA
    Samples\v5.5\0_Simple\simpleP2P../../bi
    n/win64/Release/simpleP2P.exe] - Starting... Checking for multiple
    GPUs... CUDA-capable device count: 3</p>
    
    <blockquote>
      <p>GPU0 = "   Quadro K4000" IS  capable of Peer-to-Peer (P2P)</p>
      
      <p>GPU1 = "   Quadro K2000" IS  capable of Peer-to-Peer (P2P)</p>
      
      <p>GPU2 = " GeForce GT 640" NOT capable of Peer-to-Peer (P2P)</p>
    </blockquote>
    
    <p>Checking GPU(s) for support of peer to peer memory access...</p>
    
    <blockquote>
      <p>Peer-to-Peer (P2P) access from Quadro K4000 (GPU0) -> Quadro K2000 (GPU1) : No</p>
      
      <p>Peer-to-Peer (P2P) access from Quadro K2000 (GPU1) -> Quadro K4000 (GPU0) : No</p>
    </blockquote>
    
    <p>Two or more SM 2.0 class GPUs are required for C:\ProgramData\NVIDIA
    Corporation \CUDA
    Samples\v5.5\0_Simple\simpleP2P../../bin/win64/Release/simpleP2P.exe
    to r un. Support for UVA requires a GPU with SM 2.0 capabilities. Peer
    to Peer access is not available between GPU0 &lt;-> GPU1, waiving test.</p>
  </blockquote>
  
  <p>Quadros in TCC-mode:</p>
  
  <pre><code>nvidia-smi.exe"
  Tue Mar 11 12:43:05 2014
  +------------------------------------------------------+
  | NVIDIA-SMI 5.320.57   Driver Version: 320.57         |
  |-------------------------------+----------------------+----------------------+
  | GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  |===============================+======================+======================|
  |   0  Quadro K2000        TCC  | 0000:01:00.0     Off |                  N/A |
  | 30%   30C    P8    N/A /  N/A |        6MB /  2047MB |      0%      Default |
  +-------------------------------+----------------------+----------------------+
  |   1  GeForce GT 640     WDDM  | 0000:02:00.0     N/A |                  N/A |
  | 40%   32C  N/A     N/A /  N/A |     2016MB /  2047MB |     N/A      Default |
  +-------------------------------+----------------------+----------------------+
  |   2  Quadro K4000        TCC  | 0000:03:00.0     Off |                  N/A |
  | 30%   36C    P8    10W /  87W |        8MB /  3071MB |      0%      Default |
  +-------------------------------+----------------------+----------------------+
  
  +-----------------------------------------------------------------------------+
  | Compute processes:                                               GPU Memory |
  |  GPU       PID  Process name                                     Usage      |
  |=============================================================================|
  |    1            Not Supported                                               |
  </code></pre>
  
  <p>In the documentation said that: <a href="https://developer.nvidia.com/gpudirect" rel="nofollow">https://developer.nvidia.com/gpudirect</a></p>
  
  <blockquote>
    <p><strong>GPUDirect</strong> eliminates unnecessary system memory copies, dramatically
    lowers CPU overhead, and reduces latency, resulting in significant
    performance improvements in data transfer times for applications
    running on NVIDIA Tesla™ <strong>and Quadro™ products.</strong></p>
  </blockquote>
  
  <p>More detailed specifications of Quadros there, but there are only about <a href="https://developer.nvidia.com/gpudirect" rel="nofollow">GPUDirect For Video</a>, and nothing about P2P: <a href="http://www.nvidia.com/content/PDF/line_card/6660-nv-prographicssolutions-linecard-july13-final-lr.pdf" rel="nofollow">http://www.nvidia.com/content/PDF/line_card/6660-nv-prographicssolutions-linecard-july13-final-lr.pdf</a></p>
  
  <p>About PCIe bus:</p>
  
  <pre><code>nvidia-smi -q
  GPU 0000:01:00.0
      Product Name                    : Quadro K2000
      PCI
          Bus                         : 0x01
          Device                      : 0x00
          Domain                      : 0x0000
          Device Id                   : 0x0FFE10DE
          Bus Id                      : 0000:01:00.0
          Sub System Id               : 0x094C10DE
          GPU Link Info
              PCIe Generation
                  Max                 : 2
                  Current             : 1
              Link Width
                  Max                 : 16x
                  Current             : 8x
      FB Memory Usage
          Total                       : 2047 MiB
          Used                        : 6 MiB
          Free                        : 2041 MiB
      BAR1 Memory Usage
          Total                       : 256 MiB
          Used                        : 2 MiB
          Free                        : 254 MiB
      Compute Mode                    : Default
  ...
  
  GPU 0000:02:00.0
      Product Name                    : GeForce GT 640
      PCI
          Bus                         : 0x02
          Device                      : 0x00
          Domain                      : 0x0000
          Device Id                   : 0x0FC110DE
          Bus Id                      : 0000:02:00.0
          Sub System Id               : 0x8A921462
          GPU Link Info
              PCIe Generation
                  Max                 : N/A
                  Current             : N/A
              Link Width
                  Max                 : N/A
                  Current             : N/A
  
  ...
  
  GPU 0000:03:00.0
      Product Name                    : Quadro K4000
      PCI
          Bus                         : 0x03
          Device                      : 0x00
          Domain                      : 0x0000
          Device Id                   : 0x11FA10DE
          Bus Id                      : 0000:03:00.0
          Sub System Id               : 0x097C10DE
          GPU Link Info
              PCIe Generation
                  Max                 : 2
                  Current             : 1
              Link Width
                  Max                 : 16x
                  Current             : 16x
      FB Memory Usage
          Total                       : 3071 MiB
          Used                        : 8 MiB
          Free                        : 3063 MiB
      BAR1 Memory Usage
          Total                       : 256 MiB
          Used                        : 2 MiB
          Free                        : 254 MiB
      Compute Mode                    : Default
  </code></pre>
  
  <p>Can I use GPUDirect v2 P2P with Quadros, and if I can, then in which of these?
  Should be size of the BAR1 is equal to the size of GPU-RAM to be able to use P2P?</p>
  
  <p><strong>UPDATE 11.03.2014 23:16</strong>:</p>
  
  <ol>
  <li>I <strong>can't use P2P Direct Transfers</strong> - I transfered random generated data by using <code>cudaMemcpy(gpu_ptr1, gpu_ptr0, cudaMemcpyDefault);</code> successfully with 3 GB/sec on PCIe-gen2 8x (4 GB/sec theoretically), but function copies through the host - In VisualProfiler Context1(DtoH) and Context2(HtoD).</li>
  <li>I <strong>can't use P2P Direct Access</strong> by using <code>__global__ Kernel(char *dst, char *src, size_t size) { int idx = blockIdx.x * blockDim.x + threadIdx.x; dst[idx] = src[idx]; }</code> - I get an error when use function <code>cudaDeviceEnablePeerAccess()</code> and get <code>0</code> when using <code>cudaDeviceCanAccessPeer()</code></li>
  </ol>