title: How to use NVIDIA K80?
body: <p>The machine has 4 NVIDIA K80s setuped and the outputs of <code>nvidia-smi</code> are the information of the 4 cards. Their GPU IDs are <code>0,1,2,3</code>. Each K80 have two types of GPU memory: <code>FB</code> and <code>BAR1</code>, both have 12 Gb. But the CUDA programs always use the <code>FB</code> memory, with the <code>BAR1</code> memory left be free. When the CUDA programs allocate more than 12 Gb GPU memory in each card, an error of <code>Out of memory</code> will be occurred, but the <code>BAR1</code> memory still be not used. </p>
  
  <p>How could I use the <code>BAR1</code> memory correctly in this setting?</p>
  
  <p><strong>UPDATED</strong>
  The partial output of <code>nvidia-smi</code> is</p>
  
  <pre><code>      Compute Mode:
           &lt; Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) &gt;
      &gt; Peer access from Tesla K80 (GPU0) -&gt; Tesla K80 (GPU1) : Yes
      &gt; Peer access from Tesla K80 (GPU0) -&gt; Tesla K80 (GPU2) : No
      &gt; Peer access from Tesla K80 (GPU0) -&gt; Tesla K80 (GPU3) : No
      &gt; Peer access from Tesla K80 (GPU1) -&gt; Tesla K80 (GPU0) : Yes
      &gt; Peer access from Tesla K80 (GPU1) -&gt; Tesla K80 (GPU2) : No
      &gt; Peer access from Tesla K80 (GPU1) -&gt; Tesla K80 (GPU3) : No
      &gt; Peer access from Tesla K80 (GPU2) -&gt; Tesla K80 (GPU0) : No
      &gt; Peer access from Tesla K80 (GPU2) -&gt; Tesla K80 (GPU1) : No
      &gt; Peer access from Tesla K80 (GPU2) -&gt; Tesla K80 (GPU3) : Yes
      &gt; Peer access from Tesla K80 (GPU3) -&gt; Tesla K80 (GPU0) : No
      &gt; Peer access from Tesla K80 (GPU3) -&gt; Tesla K80 (GPU1) : No
      &gt; Peer access from Tesla K80 (GPU3) -&gt; Tesla K80 (GPU2) : Yes
  </code></pre>