title: splitting a large tab delimited file in python
body: <p>I have a large tab delimited file containing about 1.4 million lines and 50 columns. Before I do anything with the data contained in the file I want to split this large file into about a few thousand smaller files. The first column of my file contains position information, and I want each smaller file to be a specific interval based on this information. In separate lists I have the start and stop of each interval that I want to split the larger file by. Here is the part of my code that does this operation, the start and stop positions are contained in lists called start_L and stop_L:</p>
  
  <pre><code>for i in range(len(id)):
     out1=((file%s.txt)%(id[i]))
     table=open('largefile.tsv',"r")
     start=int(start_L[i])
     stop=int(stop_L[i])
     table.next()
     temp_out=open(out1,"w")
     reader=csv.reader(table,delimiter="\t")
     for line in reader:
         if int(line[0]) in range(start,stop):
             for y in line:
                 temp_out.write(("%s\t")%(y))
             temp_out.write("\n")
      else:
          if int(line[0]) &gt; stop:
              break
          else:
              pass
  print "temporary file..." , id[i]
  </code></pre>
  
  <p>The above code achieves what I want, but is extremely slow. It can process the first hundred or so intervals in a matter of minutes, but gets exponentially slower with each passing interval, so it takes days to run. Is there a faster, or more efficient way of doing this? I believe the problem is that it has to scan over the entire file to find the positions within the specified interval each time through the loop. </p>