title: pthread_create fails with EAGAIN
body: <p>Consider this code snippet here, where I am trying to create a bunch of threads which end up processing a given task which simulates a race-condition.</p>
  
  <pre><code>const int thread_count = 128;
  pthread_t threads[thread_count];
  
  for (int n = 0; n != thread_count; ++n)
  {
      ret = pthread_create(&amp;threads[n], 0, test_thread_fun, &amp;test_thread_args);
      if( ret != 0 )
      {
          fprintf( stdout, "Fail %d %d", ret, errno );
          exit(0);
       }
   }
  </code></pre>
  
  <p>Things generally works fine except occasionally pthread_create fails with errno EAGAIN "resource temporarily unavailable", I tried inducing usleep, and retry creating but with no real effect.</p>
  
  <p>the failure is sporadic and on some boxes no failures and on some occurs very frequently.</p>
  
  <p>any Idea what could be going wrong here ?</p>
  
  <blockquote>
    <p>Edit - 1</p>
  </blockquote>
  
  <p>update on max-threads </p>
  
  <pre><code>cat /proc/sys/kernel/threads-max
  256467
  </code></pre>
  
  <blockquote>
    <p>Edit 2</p>
  </blockquote>
  
  <p>I think the inputs here kept me thinking, i'll probably do the below and post any results that are worth sharing.</p>
  
  <ol>
  <li>set stack size to a minimum value, I don't think thread_function uses any large arrays.</li>
  <li>increase my memory and swap ( and nullify any side effects )</li>
  <li>write a script to monitor the system behavior and see any other process/system daemon is interfering when this case is run, which is in turn can cause resource crunch.</li>
  <li>system hard and soft limits are pretty high so I'll leave them as they are at this point.</li>
  </ol>