title: netfilter_queue spurious packets
body: <p>I'm implementing a user-space firewall using the netfilter queue library. I got a file descriptor for the queue using <code>nfq_fd()</code> so I can call <code>recv(fd, recv_buf, BUFFERSIZE, MSG_DONTWAIT)</code> to get packet data without blocking. But sometimes <code>recv()</code> starts returning 52-byte packets every time I call it. If I check the output of <code>iptables -nvL INPUT</code> the number of packets doesn't increase, so they're not actually being sent from the network.  Edit3: nfq_handle_packet() returns -1 when I pass it one of these odd packets, and it never triggers the callback function, so I have no way to get a packet id or return a verdict.</p>
  
  <p>Why is recv() giving me these odd packets?</p>
  
  <p>Edit1:</p>
  
  <p>The packets aren't all identical but they have a similar structure. There are also some repeats. Here's a hexdump of a few of them:</p>
  
  <pre><code>0000   34 00 00 00 02 00 00 00  00 00 00 00 BE 4E 00 00   4............N..
  0010   FE FF FF FF 20 00 00 00  01 03 01 00 00 00 00 00   .... ...........
  0020   00 00 00 00 00 00 00 00  0C 00 02 00 00 00 00 01   ................
  0030   01 00 00 00                                        ....
  
  0000   34 00 00 00 02 00 00 00  00 00 00 00 5B 69 00 00   4...........[i..
  0010   FE FF FF FF 20 00 00 00  01 03 01 00 00 00 00 00   .... ...........
  0020   00 00 00 00 00 00 00 00  0C 00 02 00 00 00 00 01   ................
  0030   00 00 01 95                                        ....
  
  0000   34 00 00 00 02 00 00 00  00 00 00 00 5B 69 00 00   4...........[i..
  0010   FE FF FF FF 20 00 00 00  01 03 01 00 00 00 00 00   .... ...........
  0020   00 00 00 00 00 00 00 00  0C 00 02 00 00 00 00 01   ................
  0030   00 00 01 95                                        ....
  </code></pre>
  
  <p>Edit2:</p>
  
  <p>The code is pretty rudimentary, it's just tweaked from a few netfilter_queue tutorials I found.</p>
  
  <pre><code>#include &lt;linux/netfilter.h&gt;
  #include &lt;libnetfilter_queue/libnetfilter_queue.h&gt;
  #include &lt;stdlib.h&gt;
  #include &lt;stdio.h&gt;
  #include &lt;stdint.h&gt;
  #include &lt;string.h&gt;
  #include &lt;syslog.h&gt;
  
  #define BUFFERSIZE 500
  
  int main()
  {
     struct nfq_handle *h;
     struct nfq_q_handle *qh;
     struct my_nfq_data msg;
     int fd;
     unsigned char recv_buf[BUFFERSIZE];
     int action;
  
     if ((stat("/proc/net/netfilter/nfnetlink_queue", &amp;fbuf) &lt; 0) &amp;&amp; (errno == ENOENT))
       {
          fprintf(stderr, "Please make sure nfnetlink_queue is installed, or that you have\ncompiled a kernel with the Netfilter QUEUE target built in.\n");
          exit(EXIT_FAILURE);
       }
  
     openlog("packetbl", LOG_PID, "local6");
  
     if ((h = nfq_open()) == 0)
       {    
          syslog(LOG_ERR, "Couldn't open netlink connection: %s", strerror(errno));
          exit(EXIT_FAILURE);
       }    
  
     nfq_unbind_pf(h, AF_INET);
     if ((nfq_bind_pf(h, AF_INET) &lt; 0))
       {    
          syslog(LOG_ERR, "Couldn't bind to IPv4: %s", strerror(errno));
       }    
  
     nfq_unbind_pf(h, AF_INET6);
     if ((nfq_bind_pf(h, AF_INET6) &lt; 0))
       {    
          syslog(LOG_ERR, "Couldn't bind to IPv6: %s", strerror(errno));
       }    
  
     if ((qh = nfq_create_queue(h, 0, &amp;callback, &amp;msg)) == NULL)
       {    
          syslog(LOG_ERR, "Couldn't create nfq: %s", strerror(errno));
          exit(EXIT_FAILURE);
       }    
  
     if ((nfq_set_mode(qh, NFQNL_COPY_PACKET, BUFFERSIZE)) == -1)
       {    
          syslog(LOG_ERR, "nfq_set_mode error: %s", strerror(errno));
          if (errno == 111) 
            {    
               syslog(LOG_ERR, "try loading the nfnetlink_queue module");
            }    
          exit(EXIT_FAILURE);
       }    
  
     fd = nfq_fd(h);
  
     while(1)
       {
  
          /* Up here I print some statistics on packets allowed and blocked.
             It prints on a schedule, so the recv() call has to be non-blocking
             or else the statistics would only print out when there's a packet. */
  
          recv_return_code = recv(fd, recv_buf, BUFFERSIZE, MSG_DONTWAIT); //nonblocking
  
          if (recv_return_code &lt; 0)
            {
              if (errno == EAGAIN ||
                  errno == EWOULDBLOCK)
                {
                  nanosleep(&amp;times,NULL);
                }
              else
                {
                  syslog(LOG_ERR, "recv failed: %s", strerror(errno));
                }
              continue;
            }
  
          printf("received %d bytes\n", recv_return_code);
  
          /* when nfq_handle_packet() succeeds, it triggers the callback
             which puts the packet data into a global variable "msg" */
          if (nfq_handle_packet(h, recv_buf, recv_return_code) != 0)
            {
              syslog(LOG_ERR, "couldn't handle packet");
            }
  
          action = packet_check_ip(msg);
  
          pbl_set_verdict(qh, ntohl(msg.header.packet_id), action);
       }
  }
  </code></pre>
  
  <p>Edit 4:</p>
  
  <p>I'm using scapy as a traffic generator. If I send only one packet at a time, then I get 0 or 1 bogus packets and then it stops. Here's the output from strace:</p>
  
  <pre><code>recvfrom(3, "x\0\0\0\0\3\0\0\0\0\0\0\0\0\0\0\n\0\0\0\v\0\1\0\0\0\0\6\206\335\1\0\10\0\5\0\0\0\0\2\20\0\t\0\0\6\261\201\0\f)7Z\22\0\0@\0\n\0`\0\0\0\0\24\6@&amp;\6\364\0\10\0\0\0\0\0\0\0\0\0p\5&amp;\6\364\0\10\0\0\0\0\0\0\0\0\0p\4\0\24\0\31\0\0\0\0\0\0\0\0P\2 \0k\236\0\0", 9216, MSG_DONTWAIT, NULL, NULL) = 120
  sendto(4, "&lt;182&gt;Jan 13 10:51:20 packetbl[8785]: [Found in cache (accept)] [2606:f400:800::7005,20,25]", 90, MSG_NOSIGNAL, NULL, 0) = 90
  sendmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{" \0\0\0\1\3\1\0\0\0\0\0\0\0\0\0\0\0\0\0\f\0\2\0\0\0\0\1\0\0\0\6", 32}], msg_controllen=0, msg_flags=0}, 0) = 32
  recvfrom(3, "x\0\0\0\0\3\0\0\0\0\0\0\0\0\0\0\n\0\0\0\v\0\1\0\0\0\0\7\206\335\1\0\10\0\5\0\0\0\0\2\20\0\t\0\0\6\261\201\0\f)7Z\22\0\0@\0\n\0`\0\0\0\0\24\6@&amp;\6\364\0\10\0\0\0\0\0\0\0\0\0p\1&amp;\6\364\0\10\0\0\0\0\0\0\0\0\0p\4\0\24\0\31\0\0\0\0\0\0\0\0P\2 \0k\242\0\0", 9216, MSG_DONTWAIT, NULL, NULL) = 120
  futex(0x60c984, FUTEX_CMP_REQUEUE_PRIVATE, 1, 2147483647, 0x607fc0, 8) = 2
  futex(0x607fc0, FUTEX_WAKE_PRIVATE, 1)  = 1
  sendmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{" \0\0\0\1\3\1\0\0\0\0\0\0\0\0\0\0\0\0\0\f\0\2\0\0\0\0\1\7\0\0\0", 32}], msg_controllen=0, msg_flags=0}, 0) = 32
  recvfrom(3, "4\0\0\0\2\0\0\0\0\0\0\0Q\"\0\0\376\377\377\377 \0\0\0\1\3\1\0\0\0\0\0\0\0\0\0\0\0\0\0\f\0\2\0\0\0\0\1\7\0\0\0", 9216, MSG_DONTWAIT, NULL, NULL) = 52
  sendto(4, "&lt;179&gt;Jan 13 10:51:22 packetbl[8785]: couldn't handle packet", 59, MSG_NOSIGNAL, NULL, 0) = 59
  sendmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{" \0\0\0\1\3\1\0\0\0\0\0\0\0\0\0\0\0\0\0\f\0\2\0\0\0\0\1\0\0\0\7", 32}], msg_controllen=0, msg_flags=0}, 0) = 32
  </code></pre>
  
  <p>I can send individual packets as fast as I can twiddle my fingers and it never goes into a death spiral. But if I have scapy send 4 packets at once it sometimes triggers one (or zero) bogus packets for each real packet, but other times I receive infinite bogus packets. If I send lots of packets it always goes infinite.</p>
  
  <p>I had seen some of this behavior before but Nominal Animal's answer jogged my memory.  One odd thing about my code as shown above is that I still do <code>packet_check_ip()</code> and <code>pbl_set_verdict()</code> even if <code>nfq_handle_packet()</code> fails. I thought it would make sense to put a <code>continue;</code> in that case, because otherwise I'm processing stale data in the <code>msg</code> variable. (Correct me if I'm wrong, but that should have the same effect as moving the packet processing and verdict into the callback.) But that consistently sets off an infinity of bogus packets after even 1 real packet. I also moved the verdict into the callback temporarily and it didn't change anything.</p>
  
  <p>So somehow, calling set_verdict on old data sometimes prevents the infinity?</p>
  
  <p>Oh and here's the code for <code>pbl_set_verdict()</code> if anyone was worried that it might be doing anything clever :)</p>
  
  <pre><code>static void pbl_set_verdict(struct nfq_q_handle *qh,
                              uint32_t id,
                              unsigned int verdict)
  {  
     nfq_set_verdict(qh, id, verdict, 0, NULL);
  }
  </code></pre>
  
  <p>Edit 5:</p>
  
  <p>I've compiled and run the nfqnl_test.c example that is distributed with libnetfilter_queue, and it runs just fine. So it's probably not an issue with the library itself.</p>
  
  <p>Edit 6:</p>
  
  <p>Now I'm getting somewhere :) It turns out, in an over-capacity condition, <code>ntohl()</code> was being called twice! And because I was calling <code>pbl_set_verdict()</code> on the stale data even when <code>nfq_handle_packet</code> failed, it ran the data through properly which had the right effect. That's why the queue filled up when I moved the <code>pbl_set_verdict()</code> call into the callback function - it never got a chance to fix the problem caused by the over-capacity condition. And the stale data only included some of the packets processed, so a bunch of them would fill the queue eventually anyway.</p>
  
  <p>Even though my program works now, I'm still baffled about what these packets are and why they don't seem to be documented.</p>