title: Python multiprocessing and save data in order
body: <p>I'm analyzing a cvs file data line by line. For each line, I will generate a string storing the analyzed results.</p>
  
  <p>Since the file is huge, I must do multiprocessing. But because I have to access the results by their index,I have to store them in order (to a local file). </p>
  
  <p>One way I tried and worked is using lock, but it's still very slow. If i didn't lock it, it runs fast, but the data will be messed.</p>
  
  <p>What I want to do is to store those results in order to a global list. And I can write to a local file when all of the subprocesses are done.
  Any hints how to do it without a lock and how to speed it up?</p>
  
  <p>The following are my multiprocessing part code:</p>
  
  <pre><code>def worker(dat,fileName,l):
      l.acquire()
      target = open(fileName,"a")
      for values in dat:
          # recursively apply different start mean2, find best solution
          model = MixGaussian(values)
          bf = model.findBestFit()
          # find the value where it's equally probable belongs to two gaussians 
          m1 = bf[0]
          m2 = bf[1]
          d1 = bf[2]
          d2 = bf[3]
  
          # calculate x
          k = math.log((d1+0.001)/(d2+0.001))* d1 * d2
          a = d1 -d2 
          b = 2 * (m1*d2 - m2 * d1)
          c = m2 * m2 * d1 - m1* m1 * d2 - k 
          delta = -1 * math.sqrt(b*b - 4 * a * c)
          if a == 0:
              a += 0.01
          x = (-1 * b + delta) / (2 * a)
          bf.append(x)
          print bf
  
          target.write(",".join(str(ele) for ele in bf))
          target.write("\n")
  
      target.close()
      l.release()
  
  if __name__ == "__main__":
      # read from line 8000 to 8999
      data = readFile("unc_expr.tsv",8000,9000)
      target = open("modelstest9.csv","w")
      target.write("mean_1,mean_2,deviantion_1,deviation_2,cross_value")
      target.write("\n")
      target.close()
  
      numPrcs = 16
      d = []
      for i in range(numPrcs-1):
          d.append(data[i*len(data)/numPrcs:(i+1) *len(data)/numPrcs])
      d.append(data[(numPrcs-1)*len(data)/numPrcs:])
  
      start_time = time.time()
      lock = Lock()
      print("start time: %s"%start_time)
  
      for i in range(numPrcs):    
          Process(target=worker,args=(d[i],"modelstest9.csv",lock)).start()
  </code></pre>
  
  <p>Thanks!!</p>