title: Too many open files error but lsof shows a legal number of open files
body: <p>My Java program is failing with</p>
  
  <pre><code>Caused by: java.io.IOException: Too many open files
          at java.io.UnixFileSystem.createFileExclusively(Native Method)
          at java.io.File.createNewFile(File.java:883)...
  </code></pre>
  
  <p>Here are key lines from <code>/etc/security/limits.conf</code>. They set the max files for a user at 500k:</p>
  
  <pre><code>root                     soft    nofile          500000
  root                     hard    nofile          500000
  *                        soft    nofile          500000
  *                        hard    nofile          500000
  </code></pre>
  
  <p>I ran <code>lsof</code> to to count the number of files open -- both globally and by the jvm process. I examined counters in <code>/proc/sys/fs</code>. All seems OK. My process only has 4301 files open and the limit is 500k: </p>
  
  <pre><code>:~# lsof | wc -l
  5526
  :~# lsof -uusername | wc -l
  4301
  :~# cat /proc/sys/fs/file-max
  744363
  :~# cat /proc/sys/fs/file-max
  744363
  :~# cat /proc/sys/fs/file-nr
  4736    0       744363
  </code></pre>
  
  <p>This is an Ubuntu 11.04 server. I have even rebooted so I am positive these parameters are being used.</p>
  
  <p>I don't know if it's relevant, but the process is started by an upstart script, which starts the process using setuidgid, like this:</p>
  
  <pre><code>exec setuidgid username java $JAVA_OPTS -jar myprogram.jar
  </code></pre>
  
  <p>What I am missing?</p>