title: AF_UNIX socket overhead?
body: <p>I'm seeing a couple strange things with a pair of AF_UNIX sockets created by a call such as:</p>
  
  <pre><code> socketpair(AF_UNIX, SOCK_STREAM, 0, sfd); 
  </code></pre>
  
  <p>Where sfd is an int[2] array for the file descriptors.</p>
  
  <p>First, the default buffer size seems to be exactly 122K (124928 bytes), rather than anything from /proc/sys/net (such as wmem_default which is set to 128K).  Does anyone know the cause of this strange buffer size?</p>
  
  <p>Second, when writing small messages through the socket (8 bytes).  I can only write 423 of them before the write blocks, which is only 8*423 = 3384 bytes, another odd size.  The messages are acting as though they're taking up 295 + a little bytes each.  What's the source of this overhead?</p>
  
  <p>Running on RHEL6 (2.6.32, 64-bit)</p>
  
  <p>I wrote a program to try different sizes of data to compare overhead costs:</p>
  
  <pre><code>#include &lt;errno.h&gt;
  #include &lt;stdio.h&gt;
  #include &lt;stdint.h&gt;
  #include &lt;stdlib.h&gt;
  #include &lt;unistd.h&gt;
  #include &lt;sys/types.h&gt;
  #include &lt;sys/socket.h&gt;
  #include &lt;netinet/in.h&gt;
  
  #define DATA_SIZE 4
  
  void run(size_t size) {
      int sfd[2];
      if (socketpair(AF_UNIX, SOCK_STREAM, 0, sfd) == -1) {
          perror("error");
      }
  
  
      int sndbuf, sbsize = sizeof(sndbuf);
      getsockopt(sfd[0], SOL_SOCKET, SO_SNDBUF, &amp;sndbuf, (socklen_t*)&amp;sbsize);
  
      printf("Data Size: %zd\n", size);
      char buff[size];   
      size_t wrote=0;
      for (size_t ii=0; ii &lt; 32768; ii++) {
          if ((send(sfd[0], buff, size, MSG_DONTWAIT) == -1) &amp;&amp; (errno == EAGAIN)) {
              wrote = ii;
              break;
          }
      }
  
      printf("Wrote:     %zd\n", wrote);
  
      if (wrote != 0) { 
          int bpm = sndbuf/wrote;
          int oh  = bpm - size;
  
          printf("Bytes/msg: %i\n",  bpm);
          printf("Overhead:  %i\n",  oh);
          printf("\n");
      }
  
      close(sfd[0]); close(sfd[1]);
  }
  
  int main() {
      int sfd[2];
      socketpair(AF_UNIX, SOCK_STREAM, 0, sfd);
  
      int sndbuf, sbsize = sizeof(sndbuf);
      getsockopt(sfd[0], SOL_SOCKET, SO_SNDBUF, &amp;sndbuf, (socklen_t*)&amp;sbsize);
  
      printf("Buffer Size: %i\n\n", sndbuf);
      close(sfd[0]); close(sfd[1]);
  
      for (size_t ii=4; ii &lt;= 4096; ii *= 2) {
          run(ii);
      }
  }
  </code></pre>
  
  <p>Which gives:</p>
  
  <pre><code>Buffer Size: 124928
  
  Data Size: 4
  Wrote:     423
  Bytes/msg: 295
  Overhead:  291
  
  Data Size: 8
  Wrote:     423
  Bytes/msg: 295
  Overhead:  287
  
  Data Size: 16
  Wrote:     423
  Bytes/msg: 295
  Overhead:  279
  
  Data Size: 32
  Wrote:     423
  Bytes/msg: 295
  Overhead:  263
  
  Data Size: 64
  Wrote:     423
  Bytes/msg: 295
  Overhead:  231
  
  Data Size: 128
  Wrote:     348
  Bytes/msg: 358
  Overhead:  230
  
  Data Size: 256
  Wrote:     256
  Bytes/msg: 488
  Overhead:  232
  
  Data Size: 512
  Wrote:     168
  Bytes/msg: 743
  Overhead:  231
  
  Data Size: 1024
  Wrote:     100
  Bytes/msg: 1249
  Overhead:  225
  
  Data Size: 2048
  Wrote:     55
  Bytes/msg: 2271
  Overhead:  223
  
  Data Size: 4096
  Wrote:     29
  Bytes/msg: 4307
  Overhead:  211
  </code></pre>
  
  <p>Versus using a pipe there's definitely a lot of overhead:</p>
  
  <pre><code>Data Size: 4
  Wrote:     16384
  Bytes/msg: 4
  Overhead:  0
  
  Data Size: 8
  Wrote:     8192
  Bytes/msg: 8
  Overhead:  0
  
  Data Size: 16
  Wrote:     4096
  Bytes/msg: 16
  Overhead:  0
  
  Data Size: 32
  Wrote:     2048
  Bytes/msg: 32
  Overhead:  0
  
  Data Size: 64
  Wrote:     1024
  Bytes/msg: 64
  Overhead:  0
  
  Data Size: 128
  Wrote:     512
  Bytes/msg: 128
  Overhead:  0
  
  Data Size: 256
  Wrote:     256
  Bytes/msg: 256
  Overhead:  0
  
  Data Size: 512
  Wrote:     128
  Bytes/msg: 512
  Overhead:  0
  
  Data Size: 1024
  Wrote:     64
  Bytes/msg: 1024
  Overhead:  0
  
  Data Size: 2048
  Wrote:     32
  Bytes/msg: 2048
  Overhead:  0
  
  Data Size: 4096
  Wrote:     16
  Bytes/msg: 4096
  Overhead:  0
  </code></pre>