title: python multiprocessing fastq function
body: <p>I am a new user of mutliprocessing module in Python3. 
  I have 2 fastq files (forward and reverse) and I want to process forward/reverse couple of reads. For this, from forward read, I get the corresponding reverse and apply a function with multiple arguments on the couple. So far, I've done it sequentially on 1 thread, which is quite long for huge files. Now, I would like to improve speed by parallelising the function application, so I create chunk of the forward file and apply function to each chunk using multiprocessing. Here is the code:</p>
  
  <pre><code>def chunk_itr(iterator, chunk_size):
      """
      Function to split fastq file into smallest files for faster processing
      From biopython solutions
      """
      entry = True
      while entry:
          chunk = []
          while len(chunk) &lt; chunk_size:
              try:
                  entry = next(iterator)
              except StopIteration:
                  entry = None
              if entry is None:
                  break
              chunk.append(entry)
          if chunk:
              yield chunk
  
  
  def chunk_fastq(f_fastq, chunkSize, path2out):
      rec_itr = SeqIO.parse(open(f_fastq), "fastq")
      os.mkdir(os.path.join(path2out, "chunk_files"))
      dir_out = os.path.join(path2out, "chunk_files")
      base = os.path.basename(f_fastq)
      fname = os.path.splitext(base)[0]
      for i, chunk in enumerate(chunk_itr(rec_itr, chunkSize)):
          out_chunk_name = os.path.join(dir_out, "{0}_chunk{1}.fastq".format(fname, i))
          with open(out_chunk_name, "w") as handle:
              SeqIO.write(chunk, handle, "fastq")
  
  def testmulti(fwd_chunk, rev_idx):
      fwd_idx = SeqIO.index(fwd_chunk, "fastq")
      for i in fwd_idx:
          print(i, rev_idx[i])
  
  pathfwd = "path/to/forward_file"
  f_rev = "path/to/rev_fastq"
  
  def main():
      rev_idx = SeqIO.index(f_rev, "fastq")
      chunk_fastq(pathfwd, 1000, path2chunk)
      files = [f for f in os.listdir(path2chunk)]
  
  # sequential
      for i in files:
          testmulti(i, rev_idx)
  
  # parallel process
      processes = []
      for i in files:
          proc = mp.Process(target=testmulti, args=(i, rev_idx,))
          processes.append(proc)
          proc.start()
      for p in processes:
          p.join()
  </code></pre>
  
  <p>The sequential approach works fine but the parallel one crash with the following error:</p>
  
  <pre><code>Process Process-2:
  Traceback (most recent call last):
  Traceback (most recent call last):
    File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
      self.run()
    File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
      self.run()
    File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
      self._target(*self._args, **self._kwargs)
    File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
      self._target(*self._args, **self._kwargs)
    File "test.py", line 28, in testmulti
      print(i, rev_idx[i])
    File "test.py", line 28, in testmulti
      print(i, rev_idx[i])
    File "/home/user/.local/lib/python3.6/site-packages/Bio/File.py", line 417, in __getitem__
      record = self._proxy.get(self._offsets[key])
    File "/home/user/.local/lib/python3.6/site-packages/Bio/File.py", line 417, in __getitem__
      record = self._proxy.get(self._offsets[key])
    File "/home/user/.local/lib/python3.6/site-packages/Bio/SeqIO/_index.py", line 69, in get
      return self._parse(StringIO(_bytes_to_string(self.get_raw(offset))))
    File "/home/user/.local/lib/python3.6/site-packages/Bio/SeqIO/_index.py", line 69, in get
      return self._parse(StringIO(_bytes_to_string(self.get_raw(offset))))
    File "/home/user/.local/lib/python3.6/site-packages/Bio/SeqIO/_index.py", line 664, in get_raw
      raise ValueError("Problem with quality section")
    File "/home/user/.local/lib/python3.6/site-packages/Bio/SeqIO/_index.py", line 642, in get_raw
      raise ValueError("Premature end of file in seq section")
  ValueError: Problem with quality section
  ValueError: Premature end of file in seq section
  </code></pre>
  
  <p>From the Index class description in biopython, it seems there is an issue with the file format/structure
  I double checked the input files and there are no error (and it works with the sequential approach).
  My guess so far:</p>
  
  <ol>
  <li>using Process like this is not a good option (I also tried pool.starmap, but without success)</li>
  <li>since the f_rev is indexed once and then each process try to use it in parallel, there is a conflict </li>
  </ol>
  
  <p>Any help would be appreciated</p>
  
  <p>Thank you!</p>