title: MPI in Python: load data from a file by line concurrently
body: <p>I'm new to python as well as MPI.
  I have a huge data file, 10Gb, and I want to load it into, i.e., a list or whatever more efficient, please suggest.</p>
  
  <p>Here is the way I load the file content into a list</p>
  
  <pre><code>def load(source, size):
       data = [[] for _ in range(size)]
            ln = 0
            with open(source, 'r') as input:
                    for line in input:
                            ln += 1
                            data[ln%size].sanitize(line)
            return data
  </code></pre>
  
  <p>Note:</p>
  
  <ul>
  <li>source: is file name</li>
  <li>size: is the number of concurrent process, I divide data into [size] of sublist.</li>
  </ul>
  
  <p>for parallel computing using MPI in python.</p>
  
  <p>Please advise how to load data more efficient and faster. I'm searching for days but I couldn't get any results matches my purpose and if there exists, please comment with a link here.</p>
  
  <p>Regards</p>