title: Socket recv() hang on large message with MSG_WAITALL
body: <p>I have an application that reads large files from a server and hangs frequently on a particular machine.  It has worked successfully under RHEL5.2 for a long time.  We have recently upgraded to RHEL6.1 and it now hangs regularly.</p>
  
  <p>I have created a test app that reproduces the problem.  It hangs approx 98 times out of 100.</p>
  
  <pre><code>#include &lt;errno.h&gt;
  #include &lt;stdio.h&gt;
  #include &lt;stdlib.h&gt;
  #include &lt;string.h&gt;
  #include &lt;sys/param.h&gt;
  #include &lt;sys/stat.h&gt;
  #include &lt;sys/types.h&gt;
  #include &lt;unistd.h&gt;
  #include &lt;netdb.h&gt;
  #include &lt;sys/socket.h&gt;
  #include &lt;sys/time.h&gt;
  
  int mFD = 0;
  
  void open_socket()
  {
    struct addrinfo hints, *res;
    memset(&amp;hints, 0, sizeof(hints));
    hints.ai_socktype = SOCK_STREAM;
    hints.ai_family = AF_INET;
  
    if (getaddrinfo("localhost", "60000", &amp;hints, &amp;res) != 0)
    {
      fprintf(stderr, "Exit %d\n", __LINE__);
      exit(1);
    }
  
    mFD = socket(res-&gt;ai_family, res-&gt;ai_socktype, res-&gt;ai_protocol);
  
    if (mFD == -1)
    {
      fprintf(stderr, "Exit %d\n", __LINE__);
      exit(1);
    }
  
    if (connect(mFD, res-&gt;ai_addr, res-&gt;ai_addrlen) &lt; 0)
    {
      fprintf(stderr, "Exit %d\n", __LINE__);
      exit(1);
    }
  
    freeaddrinfo(res);
  }
  
  void read_message(int size, void* data)
  {
    int bytesLeft = size;
    int numRd = 0;
  
    while (bytesLeft != 0)
    {
      fprintf(stderr, "reading %d bytes\n", bytesLeft);
  
      /* Replacing MSG_WAITALL with 0 works fine */
      int num = recv(mFD, data, bytesLeft, MSG_WAITALL);
  
      if (num == 0)
      {
        break;
      }
      else if (num &lt; 0 &amp;&amp; errno != EINTR)
      {
        fprintf(stderr, "Exit %d\n", __LINE__);
        exit(1);
      }
      else if (num &gt; 0)
      {
        numRd += num;
        data += num;
        bytesLeft -= num;
        fprintf(stderr, "read %d bytes - remaining = %d\n", num, bytesLeft);
      }
    }
  
    fprintf(stderr, "read total of %d bytes\n", numRd);
  }
  
  int main(int argc, char **argv)
  {
    open_socket();
  
    uint32_t raw_len = atoi(argv[1]);
    char raw[raw_len];
  
    read_message(raw_len, raw);
  
    return 0;
  }
  </code></pre>
  
  <p>Some notes from my testing:</p>
  
  <ul>
  <li>If "localhost" maps to the loopback address 127.0.0.1, the app hangs on the call to recv() and NEVER returns.</li>
  <li>If "localhost" maps to the ip of the machine, thus routing the packets via the ethernet interface, the app completes successfully.</li>
  <li>When I experience a hang, the server sends a "TCP Window Full" message, and the client responds with a "TCP ZeroWindow" message (see image and attached tcpdump capture).  From this point, it hangs forever with the server sending keep-alives and the client sending ZeroWindow messages.  The client never seems to expand its window, allowing the transfer to complete.</li>
  <li>During the hang, if I examine the output of "netstat -a", there is data in the servers send queue but the clients receive queue is empty.</li>
  <li>If I remove the MSG_WAITALL flag from the recv() call, the app completes successfully.</li>
  <li>The hanging issue only arises using the loopback interface on 1 particular machine.  I suspect this may all be related to timing dependencies.</li>
  <li>As I drop the size  of the 'file', the likelihood of the hang occurring is reduced</li>
  </ul>
  
  <p>The source for the test app can be found here:</p>
  
  <p><a href="http://users.on.net/~carr/socket/socket_test.c">Socket test source</a></p>
  
  <p>The tcpdump capture from the loopback interface can be found here:</p>
  
  <p><a href="http://users.on.net/~carr/socket/capt.pcap">tcpdump capture</a></p>
  
  <p>I reproduce the issue by issuing the following commands:</p>
  
  <pre><code>&gt;  gcc socket_test.c -o socket_test
  &gt;  perl -e 'for (1..6000000){ print "a" }' | nc -l 60000
  &gt;  ./socket_test 6000000
  </code></pre>
  
  <p>This sees 6000000 bytes sent to the test app which tries to read the data using a single call to recv().</p>
  
  <p>I would love to hear any suggestions on what I might be doing wrong or any further ways to debug the issue.</p>