title: Minimizing copies when writing large data to a socket
body: <p>I am writing an application server that processes images (large data). I am trying to minimize copies when sending image data back to clients. The processed images I need to send to clients are in buffers obtained from jemalloc. The ways I have thought of sending the data back to the client is:</p>
  
  <p>1) Simple write call.</p>
  
  <pre><code>// Allocate buffer buf.
  // Store image data in this buffer.
  write(socket, buf, len);
  </code></pre>
  
  <p>2) I obtain the buffer through mmap instead of jemalloc, though I presume jemalloc already creates the buffer using mmap. I then make a simple call to write.</p>
  
  <pre><code>buf = mmap(file, len);  // Imagine proper options.
  // Store image data in this buffer.
  write(socket, buf, len);
  </code></pre>
  
  <p>3) I obtain a buffer through mmap like before. I then use sendfile to send the data:</p>
  
  <pre><code>buf = mmap(in_fd, len);  // Imagine proper options.
  // Store image data in this buffer.
  int rc;
  rc = sendfile(out_fd, file, &amp;offset, count);
  // Deal with rc.
  </code></pre>
  
  <p>It seems like (1) and (2) will probably do the same thing given jemalloc probably allocates memory through mmap in the first place. I am not sure about (3) though. Will this really lead to any benefits? Figure 4 on this <a href="http://www.linuxjournal.com/article/6345?page=0,1" rel="noreferrer">article</a> on Linux zero-copy methods suggests that a further copy can be prevented using sendfile: </p>
  
  <blockquote>
    <p>no data is copied into the socket buffer. Instead, only descriptors
    with information about the whereabouts and length of the data are
    appended to the socket buffer. The DMA engine passes data directly
    from the <strong>kernel buffer</strong> to the protocol engine, thus eliminating the
    remaining final copy.</p>
  </blockquote>
  
  <p>This seems like a win if everything works out. I don't know if my mmaped buffer counts as a kernel buffer though. Also I don't know when it is safe to re-use this buffer. Since the fd and length is the only thing appended to the socket buffer, I assume that the kernel actually writes this data to the socket asynchronously. If it does what does the return from sendfile signify? How would I know when to re-use this buffer?</p>
  
  <p>So my questions are:</p>
  
  <ol>
  <li>What is the fastest way to write large buffers (images in my case) to a socket? The images are held in memory.</li>
  <li>Is it a good idea to call sendfile on a mmapped file? If yes, what are the gotchas? Does this even lead to any wins?</li>
  </ol>