title: Python Multiprocessing: Only one process is running
body: <p>I am trying to spawn multiple parallel processes using the Python multiprocessing module. Basically, I did something like</p>
  
  <pre><code>pool = Pool(30)
  results = [pool.apply_async(foo, (trainData, featureVector, terms, selLabel)) for selLabel in selLabels]
  for r in results:
      tmp = r.get()
      modelFiles[tmp[0]] = tmp[1]
  </code></pre>
  
  <p>30 processes are spawned, however, it seems most of the processes have been put into sleep while only one process is actually running. Below is what I get from ps:</p>
  
  <pre><code>PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
  
  31267 74.6  2.4 7125412 6360080 pts/1 Sl+  13:06  24:25  \_ python2.6 /home/PerlModules/Python/DoOVA.py
  
  31427 27.4  2.3 6528532 6120904 pts/1 R+   13:20   5:18      \_ python2.6 /home/PerlModules/Python/DoOVA.py
  
  31428  0.0  1.3 4024724 3617016 pts/1 S+   13:20   0:00      \_ python2.6 /home/PerlModules/Python/DoOVA.py
  
  31429  0.0  1.3 4024724 3617016 pts/1 S+   13:20   0:00      \_ python2.6 /home/PerlModules/Python/DoOVA.py
  
  31430  0.0  1.3 4024724 3617016 pts/1 S+   13:20   0:00      \_ python2.6 /home/PerlModules/Python/DoOVA.py
  </code></pre>
  
  <p><code>DoOVA.py</code> is the script I am running. Most of them have a status <code>S+</code>.</p>
  
  <p>Could anyone give me some clue about what's the problem? I know the input arguement <code>featureVector</code> is pretty large in size, say around 300MB. Would that be a problem? The machine I am running on have several TB of memory though.</p>
  
  <p>foo does something like:</p>
  
  <pre><code>def foo(trainData, featureVector, terms, selLabel, penalty):
      outputFile = 'train_'+selLabel+'.dat'
      annotation = dict()
      for id in trainData:
          if trainData[id] == selLabel:
              annotation[id] = '1'
          else:
              annotation[id] = '-1'
      try:
          os.mkdir(selLabel)
          os.chdir(selLabel)
      except OSError:
          os.chdir(selLabel)
      ###Some more functions, which involves a command line call through call from subprocess module
      os.chdir('../')
      return (selLabel, 'SVM_' + selLabel + '.model')
  </code></pre>
  
  <p>All other input arguments are small in size. And the machine has at least 100 cpus. In every run, it takes the script a long time even before any directory was created, although there is no significant computation happened in foo before os.mkdir()</p>