title: Call to ioctl() with FIONREAD results in strange side-effects in apparent race condition,
body: <p>I'm writing a parallel neural network simulator and I've recently come across a problem in my code that totally confounds me (granted I'm only an intermediate C++ programmer so maybe I'm missing something obvious?),... My code involves a 'server' and many clients (workers) which take work from, and return results to the server. Here is the server part:</p>
  
  <pre><code>#include &lt;iostream&gt;
  #include &lt;fstream&gt;
  
  #include &lt;arpa/inet.h&gt;
  #include &lt;sys/epoll.h&gt;
  #include &lt;errno.h&gt;
  
  #include &lt;sys/ioctl.h&gt;
  
  void advanceToNextInputValue(std::ifstream &amp;trainingData, char &amp;nextCharacter)
     {
  
        nextCharacter = trainingData.peek();
        while(nextCharacter != EOF &amp;&amp; !isdigit(nextCharacter))
           {
  sleep(1);
              trainingData.get();
  sleep(1);
              nextCharacter = trainingData.peek();
           }
     }
  
  int main()
     {
        // Create a socket,...
        int listenerSocketNum = socket(AF_INET, SOCK_STREAM, 0);
  
        // Name the socket,...
        sockaddr_in socketAddress;
        socklen_t socketAddressLength = sizeof(socketAddress);
  
        inet_pton(AF_INET, "127.0.0.1", &amp;(socketAddress.sin_addr));
        socketAddress.sin_port = htons(9988);
        bind(listenerSocketNum, reinterpret_cast&lt;sockaddr*&gt;(&amp;socketAddress), socketAddressLength);
  
        // Create a connection queue for worker processes waiting to connect to this server,...
        listen(listenerSocketNum, SOMAXCONN);
  
  
        int epollInstance = epoll_create(3); // Expected # of file descriptors to monitor
  
        // Allocate a buffer to store epoll events returned from the network layer
        epoll_event* networkEvents = new epoll_event[3];
  
        // Add the server listener socket to the list of file descriptors monitored by epoll,...
        networkEvents[0].data.fd = -1; // A marker returned with the event for easy identification of which worker process event belongs to
        networkEvents[0].events = EPOLLIN | EPOLLET; // epoll-IN- since we only expect incoming data on this socket (ie: connection requests from workers),...
                                                     // epoll-ET- indicates an Edge Triggered watch
        epoll_ctl(epollInstance, EPOLL_CTL_ADD, listenerSocketNum, &amp;networkEvents[0]);
  
  
        char nextCharacter = 'A';
        std::ifstream trainingData;
  
        // General multi-purpose/multi-use variables,...
        long double v;
        signed char w;
        int x = 0;
        int y;
  
        while(1)
           {
              y = epoll_wait(epollInstance, networkEvents, 3, -1); // the -1 tells epoll_wait to block indefinitely
  
              while(y &gt; 0)
                 {
                    if(networkEvents[y-1].data.fd == -1) // We have a notification on the listener socket indicating a request for a new connection (and we expect only one for this testcase),...
                       {
                          x = accept(listenerSocketNum,reinterpret_cast&lt;sockaddr*&gt;(&amp;socketAddress), &amp;socketAddressLength);
  
                          networkEvents[y-1].data.fd = x; // Here we are just being lazy and re-using networkEvents[y-1] temporarily,...
                          networkEvents[y-1].events = EPOLLIN | EPOLLET;
  
                          // Add the socket for the new worker to the list of file descriptors monitored,...
                          epoll_ctl(epollInstance, EPOLL_CTL_ADD, x, &amp;networkEvents[y-1]);
  
                          trainingData.open("/tmp/trainingData.txt");
                       }
                    else if(networkEvents[y-1].data.fd == x) // Worker is waiting to receive datapoints for calibration,...
                       {
                          std::cout &lt;&lt; "nextCharacter before call to ioctl: " &lt;&lt; nextCharacter &lt;&lt; std::endl;
                          ioctl(networkEvents[y-1].data.fd, FIONREAD, &amp;w);
                          std::cout &lt;&lt; "nextCharacter after call to ioctl: " &lt;&lt; nextCharacter &lt;&lt; std::endl;
  
                          recv(networkEvents[y-1].data.fd, &amp;v, sizeof(v), MSG_DONTWAIT); // Retrieve and discard the 'tickle' from worker
  
                          if(nextCharacter != EOF)
                             {
                                trainingData &gt;&gt; v;
  
                                send(networkEvents[y-1].data.fd, &amp;v, sizeof(v), MSG_DONTWAIT);
                                advanceToNextInputValue(trainingData, nextCharacter);
                             }
                       }
  
                    y--;
                 }
           }
  
        close(epollInstance);
        return 0;
     }
  </code></pre>
  
  <p>And here is the client part:</p>
  
  <pre><code>#include &lt;arpa/inet.h&gt;
  
  int main()
     {
        int workerSocket = socket(AF_INET, SOCK_STREAM, 0);
  
        // Name the socket as agreed with the server:
        sockaddr_in serverSocketAddress;
        serverSocketAddress.sin_family = AF_INET;
        serverSocketAddress.sin_port = htons(9988);
        inet_pton(AF_INET, "127.0.0.1", &amp;(serverSocketAddress.sin_addr));
  
        // Connect your socket to the server's socket:
        connect(workerSocket, reinterpret_cast&lt;sockaddr*&gt;(&amp;serverSocketAddress), sizeof(serverSocketAddress));
  
        long double z;
        while(1)
           {
              send(workerSocket, &amp;z, sizeof(z), MSG_DONTWAIT); // Send a dummy result/tickle to server,...
              recv(workerSocket, &amp;z, sizeof(z), MSG_WAITALL);
           }
     }
  </code></pre>
  
  <p>The section of code I'm having trouble with is the following (from the server):</p>
  
  <pre><code>std::cout &lt;&lt; "nextCharacter before call to ioctl: " &lt;&lt; nextCharacter &lt;&lt; std::endl;
  ioctl(networkEvents[y-1].data.fd, FIONREAD, &amp;w);
  std::cout &lt;&lt; "nextCharacter after call to ioctl: " &lt;&lt; nextCharacter &lt;&lt; std::endl;
  </code></pre>
  
  <p>Here (at least on my system), under certain circumstances the call to ioctl basically wipes out the value of 'nextCharacter' and I can't figure out how or why!</p>
  
  <p>These are the results I expect to get:</p>
  
  <pre><code>$ ./server.exe
  nextCharacter before call to ioctl: A
  nextCharacter after call to ioctl: A
  nextCharacter before call to ioctl: 1
  nextCharacter after call to ioctl: 1
  nextCharacter before call to ioctl: 9
  nextCharacter after call to ioctl: 9
  nextCharacter before call to ioctl: 2
  nextCharacter after call to ioctl: 2
  nextCharacter before call to ioctl: 1
  nextCharacter after call to ioctl: 1
  nextCharacter before call to ioctl: 1
  nextCharacter after call to ioctl: 1
  nextCharacter before call to ioctl: 1
  nextCharacter after call to ioctl: 1
  nextCharacter before call to ioctl: 2
  nextCharacter after call to ioctl: 2
  nextCharacter before call to ioctl: ÿ
  nextCharacter after call to ioctl: ÿ
  </code></pre>
  
  <p>(Lowercase 'y' with umlaut is the end-of-file character EOF)</p>
  
  <p>And these are the results I do get (note that we end up in an endless loop because the stopping condition relies on the value of nextCharacter and that is wiped out, so it never stops):</p>
  
  <pre><code>$ ./server.exe
  nextCharacter before call to ioctl: A
  nextCharacter after call to ioctl:
  nextCharacter before call to ioctl: 1
  nextCharacter after call to ioctl:
  nextCharacter before call to ioctl: 9
  nextCharacter after call to ioctl:
  nextCharacter before call to ioctl: 2
  nextCharacter after call to ioctl:
  nextCharacter before call to ioctl: 1
  nextCharacter after call to ioctl:
  nextCharacter before call to ioctl: 1
  nextCharacter after call to ioctl:
  nextCharacter before call to ioctl: 1
  nextCharacter after call to ioctl:
  nextCharacter before call to ioctl: 2
  nextCharacter after call to ioctl:
  nextCharacter before call to ioctl: ÿ
  nextCharacter after call to ioctl:
  nextCharacter before call to ioctl: ÿ
  nextCharacter after call to ioctl:
  nextCharacter before call to ioctl: ÿ
  nextCharacter after call to ioctl:
  .
  .
  .
  </code></pre>
  
  <p>If I comment out any of the sleep statements in this section (in the server):</p>
  
  <pre><code>void advanceToNextInputValue(std::ifstream &amp;trainingData, char &amp;nextCharacter)
     {
  
        nextCharacter = trainingData.peek();
        while(nextCharacter != EOF &amp;&amp; !isdigit(nextCharacter))
           {
  sleep(1);
              trainingData.get();
  sleep(1);
              nextCharacter = trainingData.peek();
           }
     }
  </code></pre>
  
  <p>I then get the results I expect to get,...</p>
  
  <p>This is the makefile I am using:</p>
  
  <pre><code>$ cat Makefile
  all: server client
  
  server: server.cpp
          g++ server.cpp -o server.exe -ansi -fno-elide-constructors -O3 -pedantic-errors -Wall -Wextra -Winit-self -Wold-style-cast -Woverloaded-virtual -Wuninitialized -Winit-self
  
  client: client.cpp
          g++ client.cpp -o client.exe -ansi -fno-elide-constructors -O3 -pedantic-errors -Wall -Wextra -Winit-self -Wold-style-cast -Woverloaded-virtual -Wuninitialized -Winit-self
  </code></pre>
  
  <p>With a trainingData.txt file that looks like this:</p>
  
  <pre><code>$ cat trainingData.txt
  15616.16993666375,15616.16993666375,9.28693983312753E20,24.99528974548316,16.91935342923897,16.91935342923897,1.386594632397968E6,2.567209162871251
  </code></pre>
  
  <p>So have I discovered a new bug or am I just stupid? :) Honestly I can't see why a call to ioctl with FIONREAD which is supposed to tell me how many bytes I have on the socket waiting to be read, should in any way affect the value of the variable 'nextCharacter',...</p>
  
  <p>Note that this is a whittled down version of the original program which still manages to reproduce the problem (at least on my system), so please keep in mind that some things may not make sense in the code snippets above :)</p>
  
  <p>Terry</p>