title: Pinning user space buffer for DMA from Linux kernel
body: <p>I'm writing driver for devices that produce around 1GB of data per second. Because of that I decided to map user buffer allocated by application directly for DMA instead of copying through intermediate kernel buffer.</p>
  
  <p>The code works, more or less. But during long-run stress testing I see kernel oops with "bad page state" initiated by unrelated applications (for instance <code>updatedb</code>), probably when kernel wants to swap some pages:</p>
  
  <pre><code>[21743.515404] BUG: Bad page state in process PmStabilityTest  pfn:357518
  [21743.521992] page:ffffdf844d5d4600 count:19792158 mapcount:0 mapping:          (null) index:0x12b011e012d0132
  [21743.531829] flags: 0x119012c01220124(referenced|lru|slab|reclaim|uncached|idle)
  [21743.539138] raw: 0119012c01220124 0000000000000000 012b011e012d0132 012e011e011e0111
  [21743.546899] raw: 0000000000000000 012101300131011c 0000000000000000 012101240123012b
  [21743.554638] page dumped because: page still charged to cgroup
  [21743.560383] page-&gt;mem_cgroup:012101240123012b
  [21743.564745] bad because of flags: 0x120(lru|slab)
  [21743.569555] BUG: Bad page state in process PmStabilityTest  pfn:357519
  [21743.576098] page:ffffdf844d5d4640 count:18219302 mapcount:18940179 mapping:          (null) index:0x0
  [21743.585318] flags: 0x0()
  [21743.587859] raw: 0000000000000000 0000000000000000 0000000000000000 0116012601210112
  [21743.595599] raw: 0000000000000000 011301310127012f 0000000000000000 012f011d010d011a
  [21743.603336] page dumped because: page still charged to cgroup
  [21743.609108] page-&gt;mem_cgroup:012f011d010d011a
  ...
  Entering kdb (current=0xffff8948189b2d00, pid 6387) on processor 6 Oops: (null)
  due to oops @ 0xffffffff9c87f469
  CPU: 6 PID: 6387 Comm: updatedb.mlocat Tainted: G    B      OE   4.10.0-42-generic #46~16.04.1-Ubuntu
  ...
  </code></pre>
  
  <hr>
  
  <p>Details:</p>
  
  <p>The user buffer consists of frames and neither the buffer not the frames are page-aligned. The frames in buffer are used in circular manner for "infinite" live data transfers. For each frame I get memory pages via <code>get_user_pages_fast</code>, then convert it to scatter-gatter table with <code>sg_alloc_table_from_pages</code> and finally map for DMA using <code>dma_map_sg</code>.</p>
  
  <p>I rely on <code>sg_alloc_table_from_pages</code> to bind consecutive pages into one DMA descriptor to reduce size of S/G table sent to device. Devices are custom built and utilize FPGA. I took inspiration from many drivers doing similar mapping, especially video drivers i915 and radeon, but no one has all the stuff on one place so I might overlook something.</p>
  
  <p>Related functions (<code>pin_user_buffer</code> and <code>unpin_user_buffer</code> are called upon separate IOCTLs):</p>
  
  <pre><code>static int pin_user_frame(struct my_dev *cam, struct udma_frame *frame)
  {
          const unsigned long bytes = cam-&gt;acq_frame_bytes;
          const unsigned long first =
                  ( frame-&gt;uaddr              &amp;  PAGE_MASK) &gt;&gt; PAGE_SHIFT;
          const unsigned long last =
                  ((frame-&gt;uaddr + bytes - 1) &amp;  PAGE_MASK) &gt;&gt; PAGE_SHIFT;
          const unsigned long offset =
                    frame-&gt;uaddr              &amp; ~PAGE_MASK;
          int nr_pages = last - first + 1;
          int err;
          int n;
          struct page **pages;
          struct sg_table *sgt;
  
          if (frame-&gt;uaddr + bytes &lt; frame-&gt;uaddr) {
                  pr_err("%s: attempted user buffer overflow!\n", __func__);
                  return -EINVAL;
          }
  
          if (bytes == 0) {
                  pr_err("%s: user buffer has zero bytes\n", __func__);
                  return -EINVAL;
          }
  
          pages = kcalloc(nr_pages, sizeof(*pages), GFP_KERNEL | __GFP_ZERO);
          if (!pages) {
                  pr_err("%s: can't allocate udma_frame.pages\n", __func__);
                  return -ENOMEM;
          }
  
          sgt = kzalloc(sizeof(*sgt), GFP_KERNEL);
          if (!sgt) {
                  pr_err("%s: can't allocate udma_frame.sgt\n", __func__);
                  err = -ENOMEM;
                  goto err_alloc_sgt;
          }
  
          /* (rw == READ) means read from device, write into memory area */
          err = get_user_pages_fast(frame-&gt;uaddr, nr_pages, READ == READ, pages);
          if (err &lt; nr_pages) {
                  nr_pages = err;
                  if (err &gt; 0) {
                          pr_err("%s: can't pin all %d user pages, got %d\n",
                                 __func__, nr_pages, err);
                          err = -EFAULT;
                  } else {
                          pr_err("%s: can't pin user pages\n", __func__);
                  }
                  goto err_get_pages;
          }
  
          for (n = 0; n &lt; nr_pages; ++n)
                  flush_dcache_page(pages[n]); //&lt;--- Is this needed?
  
          err = sg_alloc_table_from_pages(sgt, pages, nr_pages, offset, bytes,
                                          GFP_KERNEL);
          if (err) {
                  pr_err("%s: can't build sg_table for %d pages\n",
                         __func__, nr_pages);
                  goto err_alloc_sgt2;
          }
  
          if (!dma_map_sg(&amp;cam-&gt;pci_dev-&gt;dev, sgt-&gt;sgl, sgt-&gt;nents, DMA_FROM_DEVICE)) {
                  pr_err("%s: can't map %u sg_table entries for DMA\n",
                         __func__, sgt-&gt;nents);
                  err = -ENOMEM;
                  goto err_dma_map;
          }
  
          frame-&gt;pages = pages;
          frame-&gt;nr_pages = nr_pages;
          frame-&gt;sgt = sgt;
  
          return 0;
  
  err_dma_map:
          sg_free_table(sgt);
  
  err_alloc_sgt2:
  err_get_pages:
          for (n = 0; n &lt; nr_pages; ++n)
                  put_page(pages[n]);
          kfree(sgt);
  
  err_alloc_sgt:
          kfree(pages);
  
          return err;
  }
  
  static void unpin_user_frame(struct my_dev *cam, struct udma_frame *frame)
  {
          int n;
  
          dma_unmap_sg(&amp;cam-&gt;pci_dev-&gt;dev, frame-&gt;sgt-&gt;sgl, frame-&gt;sgt-&gt;nents,
                       DMA_FROM_DEVICE);
  
          sg_free_table(frame-&gt;sgt);
          kfree(frame-&gt;sgt);
          frame-&gt;sgt = NULL;
  
          for (n = 0; n &lt; frame-&gt;nr_pages; ++n) {
                  struct page *page = frame-&gt;pages[n];
                  set_page_dirty_lock(page);
                  mark_page_accessed(page); //&lt;--- Without this the Oops are more frequent
                  put_page(page);
          }
          kfree(frame-&gt;pages);
          frame-&gt;pages = NULL;
  
          frame-&gt;nr_pages = 0;
  }
  
  static void unpin_user_buffer(struct my_dev *cam)
  {
          if (cam-&gt;udma_frames) {
                  int n;
                  for (n = 0; n &lt; cam-&gt;udma_frame_count; ++n)
                          unpin_user_frame(cam, &amp;cam-&gt;udma_frames[n]);
                  kfree(cam-&gt;udma_frames);
                  cam-&gt;udma_frames = NULL;
          }
          cam-&gt;udma_frame_count = 0;
          cam-&gt;udma_buffer_bytes = 0;
          cam-&gt;udma_buffer = NULL;
          cam-&gt;udma_desc_count = 0;
  }
  
  static int pin_user_buffer(struct my_dev *cam)
  {
          int err;
          int n;
          const u32 acq_frame_count = cam-&gt;acq_buffer_bytes / cam-&gt;acq_frame_bytes;
          struct udma_frame *udma_frames;
          u32 udma_desc_count = 0;
  
          if (!cam-&gt;acq_buffer) {
                  pr_err("%s: user buffer is NULL!\n", __func__);
                  return -EFAULT;
          }
  
          if (cam-&gt;udma_buffer == cam-&gt;acq_buffer
              &amp;&amp; cam-&gt;udma_buffer_bytes == cam-&gt;acq_buffer_bytes
              &amp;&amp; cam-&gt;udma_frame_count == acq_frame_count)
                  return 0;
  
          if (cam-&gt;udma_buffer)
                  unpin_user_buffer(cam);
  
          udma_frames = kcalloc(acq_frame_count, sizeof(*udma_frames),
                                GFP_KERNEL | __GFP_ZERO);
          if (!udma_frames) {
                  pr_err("%s: can't allocate udma_frame array for %u frames\n",
                         __func__, acq_frame_count);
                  return -ENOMEM;
          }
  
          for (n = 0; n &lt; acq_frame_count; ++n) {
                  struct udma_frame *frame = &amp;udma_frames[n];
                  frame-&gt;uaddr =
                          (unsigned long)(cam-&gt;acq_buffer + n * cam-&gt;acq_frame_bytes);
                  err = pin_user_frame(cam, frame);
                  if (err) {
                          pr_err("%s: can't pin frame %d (out of %u)\n",
                                 __func__, n + 1, acq_frame_count);
                          for (--n; n &gt;= 0; --n)
                                  unpin_user_frame(cam, frame);
                          kfree(udma_frames);
                          return err;
                  }
                  udma_desc_count += frame-&gt;sgt-&gt;nents; /* Cannot overflow */
          }
          pr_debug("%s: total udma_desc_count=%u\n", __func__, udma_desc_count);
  
          cam-&gt;udma_buffer = cam-&gt;acq_buffer;
          cam-&gt;udma_buffer_bytes = cam-&gt;acq_buffer_bytes;
          cam-&gt;udma_frame_count = acq_frame_count;
          cam-&gt;udma_frames = udma_frames;
          cam-&gt;udma_desc_count = udma_desc_count;
  
          return 0;
  }
  </code></pre>
  
  <p>Related structures:</p>
  
  <pre><code>struct udma_frame {
          unsigned long   uaddr;      /* User address of the frame */
          int             nr_pages;   /* Nr. of pages covering the frame */
          struct page     **pages;    /* Actual pages covering the frame */
          struct sg_table *sgt;       /* S/G table describing the frame */
  };
  
  struct my_dev {
          ...
          u8 __user   *acq_buffer;   /* User-space buffer received via IOCTL */
          ...
          u8 __user   *udma_buffer;       /* User-space buffer for image */
          u32         udma_buffer_bytes;  /* Total image size in bytes */
          u32         udma_frame_count;   /* Nr. of items in udma_frames */
          struct udma_frame
                      *udma_frames;       /* DMA descriptors per frame */
          u32         udma_desc_count;    /* Total nr. of DMA descriptors */
          ...
  };
  </code></pre>
  
  <p>Questions:</p>
  
  <ol>
  <li>How to properly pin user buffer pages and mark them as not movable?</li>
  <li>If one frame ends and next frame starts in the same page, is it correct to handle it as two independent pages, i.e. pin the page twice?</li>
  <li>The data comes from device to user buffer and app is supposed to not write to its buffer, but I have no control over it. Can I use <code>DMA_FROM_DEVICE</code> or rather 
  use <code>DMA_BIDIRECTIONAL</code> just in case?</li>
  <li>Do I need to use something like <code>SetPageReserved</code>/<code>ClearPageReserved</code> or <code>mark_page_reserved</code>/<code>free_reserved_page</code>?</li>
  <li>Is IOMMU/swiotlb somehow involved? E.g. i915 driver doesn't use <code>sg_alloc_table_from_pages</code> if swiotlb is active?</li>
  <li>What the difference between <code>set_page_dirty</code>, <code>set_page_dirty_lock</code> and <code>SetPageDirty</code> functions?</li>
  </ol>
  
  <p>Thanks for any hint.</p>
  
  <p>PS: I cannot change the way the application gets the data without breaking our library API maintained for many years. So please do not advise e.g. to <code>mmap</code> kernel buffer...</p>