title: how to do mmap for cacheable PCIe BAR
body: <p>I am trying to write a driver with custom <code>mmap()</code> function for PCIe BAR, with the goal to make this BAR cacheable in the processor cache. I am aware this is not the best way to achieve highest bandwidth and that the order of writes is unpredictable (neither are the issues in this case).</p>
  
  <p>This is similar to what is described in <a href="https://stackoverflow.com/questions/9890728/how-would-one-prevent-mmap-from-caching-values">How would one prevent MMAP from caching values?</a></p>
  
  <p>The processor is Sandy Bridge i7, PCIe device is Altera Stratix IV dev. board.</p>
  
  <p>First, I tried to do it on CentOS 5 (2.6.18). I changed the MTRR settings to make sure the BAR is not within uncacheable MTRR and used <code>io_remap_pfn_range()</code> with <code>_PAGE_PCD</code> and <code>_PAGE_PWT</code> bits cleared. Reads worked as expected: reads returned correct values and second read to the same address does not necessarily cause the read to go to PCIe (read counter was checked in FPGA). However, the writes caused the system to freeze and then reboot without any messages in the logs or on the screen.</p>
  
  <p>Second, I tried to do it on CentOS 6 (2.6.32), which has PAT support. The result is the same: reads work correctly, writes cause system freeze and reboot. Interestingly, non-temporal/write-combining full cache line writes (AVX/SSE) work as expected, i.e. they always go to FPGA and FPGA observes full cache line writes, reads return correct values afterwards. However, simple 64-bit writes still cause system freeze/reboot.</p>
  
  <p>I also tried to <code>ioremap_cache()</code> and then <code>iowrite32()</code> inside the driver code. The result is the same.</p>
  
  <p>I think it is a hardware issue but I would appreciate if somebody can share any ideas about what's going on.</p>
  
  <p>EDIT: I was able to capture MCE message on CentOS 6: Machine Check Exception: 5 Bank 5: be2000000003110a.</p>
  
  <p>I also tried the same code on 2-socket Sandy Bridge (Romley): reads and non-temporal write behavior is the same, simple writes do not cause MCE/crash but have no effect on system state, i.e. value in memory does not change.</p>
  
  <p>Also, I tried the same code on older 2-socket Nehalem system: simple writes also cause MCE, although the codes are different.</p>