title: From a large text file to a sparse matrix with Python
body: <p>I am trying to find an efficient way to read a very large text file (about 2,000,000 lines). About 90% of these lines (the last 90% actually) have a three-column format and are used for storing a sparse matrix.</p>
  
  <p>Here is what I did. First of all, I deal with the first 10% of the file:</p>
  
  <pre><code>i=1
  cpt=0
  skip=0
  finnum=0
  indice=1 
  vec=[]
  mat=[]
  for line in fileinput.input("MY_TEXT_FILE.TXT"):
  if i==1:
      # skipping the first line
      skip = 1
  if (finnum == 0)and(skip==0):
      # special reading operation for the first 10% (approximately)
      tline=shlex.split(line)
      ind_loc=0
      while ind_loc&lt;len(tline):
      if (int(tline[ind_loc])!=0):
              vec.append(int(tline[ind_loc]))
          ind_loc=ind_loc+1   
  if (finnum == 1)and(skip==0):
      print('finnum = 1')
      h=input()    
          break       
      if (' 0' in line):
      finnum = 1
  if skip == 0:
      i=i+1
  else:
      skip=0
      i=i+1
  cpt=cpt+1
  </code></pre>
  
  <p>Then I extract the remaining 90% into a list:</p>
  
  <pre><code>matrix=[]
  with open('MY_TEXT_FILE.TXT') as f:
  for i in range(cpt):
      f.next()
  for line in f:
      matrix.append(line)
  </code></pre>
  
  <p>This allows for a very fast read through of the text file with low memory consumption. The drawback is that <strong>matrix</strong> is a list of strings, each string being something like:</p>
  
  <pre><code>&gt;&gt;&gt; matrix[23]
  '           5          11  8.320234929063493E-008\n'
  </code></pre>
  
  <p>I have tried to use an iterative procedure over the lines of <strong>matrix</strong> combined with the <strong>shlex.split</strong> command to go from a list of strings to an array but this is extremely time consuming. </p>
  
  <p>Would you be aware of fast strategies to go from a list of strings to an array ? </p>
  
  <p>What I would like to know is if there is something faster than this procedure :</p>
  
  <pre><code>A=[0]*len(matrix)
  B=[0]*len(matrix)
  C=[0]*len(matrix)
  for i in range(len(matrix)):
       line = shlex.split(matrix[i])
       A[i]=float(line[0])
       B[i]=float(line[1])
       C[i]=float(line[2])
  </code></pre>
  
  <p>Alain</p>