title: remove rows which have a specific attribute less than or equal to 0
body: <p>I have several large text files (30m+ lines, >1GB) which are being processed in ArcGIS after splitting (see <a href="https://stackoverflow.com/questions/9473921/remove-specific-lines-from-a-large-text-file-in-python">Remove specific lines from a large text file in python</a> and <a href="https://stackoverflow.com/questions/9626842/chunk-a-text-database-into-n-equal-blocks-and-retain-header">chunk a text database into N equal blocks and retain header</a> for background).</p>
  
  <p>Even after splitting the process takes over 3 days so I want to delete all xy points which have a (Rx) value less than or equal to 0. </p>
  
  <p>I haven't got python to work on reading txt datasets over 500Mb so I have used cygwin/SED commands to do the initial cleaning of the data and then python to chunk the file. So ideally the process would be to add some code to the python (see below) to not include all lines with Rx&lt;=0.</p>
  
  <pre><code>Latitude    Longitude   Rx  Best_Unit
  -16.37617    144.68805  -012.9  7
  -16.37617    144.68834  -015.1  7
  -16.37617    144.68861  -017.2  7
  -16.37617    144.68890  -018.1  7
  -16.37617    144.68919  -025.0  7
  -16.37617    144.68945  -019.5  7
  -16.37617    144.68974  -020.0  7
  -16.37617    144.69003  -020.4  7
  -16.37617    144.69623   015.3  7
  -16.37617    144.69652   015.6  7
  -16.37617    144.69679   015.8  7
  -16.37617    144.69708   016.0  7
  -16.37617    144.70076   005.0  7
  -16.37617    144.70103   002.2  7
  -16.37617    144.70131  -000.2  7
  -16.37617    144.70160  -001.5  7
  -16.37617    144.70187  -001.0  7
  -16.37617    144.70216   000.7  7
  -16.37617    144.70245   002.2  7
  -16.37617    144.70273   008.4  7
  -16.37617    144.70300   017.1  7
  -16.37617    144.70329   017.2  7
  </code></pre>
  
  <p>I want all rows (lines) where Rx>0 to be written into a new text file. I also want the column Best_Unit deleted.</p>
  
  <pre><code>from itertools import islice
  
  import arcpy, os
  #fc = arcpy.GetParameter(0)
  #chunk_size = arcpy.GetParameter(1) # number of records in each dataset
  
  fc='cb_vhn007_5.txt'
  Name = fc[:fc.rfind('.')]
  fl = Name+'.txt'
  
  headers_count = 1
  chunk_size = 500000
  
  with open(fl) as fin:
    headers = list(islice(fin, headers_count))
  
    part = 1
    while True:
      line_iter = islice(fin, chunk_size)
      try:
        first_line = line_iter.next()
      except StopIteration:
        break
      with open(Name+'_%d.txt' % part, 'w') as fout:
        for line in headers:
          fout.write(line)
        fout.write(first_line)
        for line in line_iter:
           ## add something here to check if value after third tab
           ## is &gt;0 and if so then write the row or skip.
          fout.write(line) 
  
      print "Created part %d" % part
      part += 1
  </code></pre>
  
  <p>New Code - first line includes - Rx values.</p>
  
  <pre><code>from itertools import islice
  
  import arcpy, os
  #fc = arcpy.GetParameter(0)
  #chunk_size = arcpy.GetParameter(1) # number of records in each dataset
  
  fc='cb_vhn007_5.txt'
  Name = fc[:fc.rfind('.')]
  fl = Name+'.txt'
  
  headers_count = 1
  chunk_size = 500000
  
  with open(fl) as fin:
    headers = list(islice(fin, headers_count))
  
    part = 1
    while True:
      line_iter = islice(fin, chunk_size)
      try:
        first_line = line_iter.next()
      except StopIteration:
        break
      with open(Name+'_%d.txt' % part, 'w') as fout:
        for line in headers:
          fout.write(line)
        fout.write(first_line)
        for line in line_iter:
          if line.split()[2][0:1] != '-':
            #print line.split()[2]
            fout.write(line)
  
      print "Created part %d" % part
      part += 1
  </code></pre>