How can I speed up this really basic python script for offsetting lines of numbers
Question: I have a simple text file which contains numbers in ASCII text separated by
spaces as per this example.

    
    
    150604849   
    319865.301865 5810822.964432 -96.425797 -1610
    319734.172256 5810916.074753 -52.490280 -122
    319730.912949 5810918.098465 -61.864395 -171
    319688.240891 5810889.851608 -0.339890 -1790
    *<continues like this for millions of lines>*
    

basically I want to copy the first line as is, then for all following lines I
want to offset the first value (x), offset the second value (y), leave the
third value unchanged and offset and half the last number.

I've cobbled together the following code as a python learning experience
(apologies if it crude and offensive, truly I mean no offence) and it works
ok. However the input file I'm using it on is several GB in size and I'm
wondering if there's ways to speed up the execution. Currently for a 740 MB
file it takes 2 minutes 21 seconds

    
    
    import glob
    
    #offset values
    offsetx = -306000
    offsety = -5806000
    
    files = glob.glob('*.pts')
    for file in files:
        currentFile = open(file, "r")
        out = open(file[:-4]+"_RGB_moved.pts", "w")
        firstline = str(currentFile.readline())
        out.write(str(firstline.split()[0]))
    
        while 1:
            lines = currentFile.readlines(100000)
            if not lines:
                break
            for line in lines:
                out.write('\n')
                words = line.split()
                newwords = [str(float(words[0])+offsetx), str(float(words[1])+offsety), str(float(words[2])), str((int(words[3])+2050)/2)]              
                out.write(" ".join(newwords))
    

Many thanks


Answer: Don't use `.readlines()`. Use the file directly as an iterator:

    
    
    for file in files:
        with open(file, "r") as currentfile, open(file[:-4]+"_RGB_moved.pts", "w") as out:
            firstline = next(currentFile)
            out.write(firstline.split(None, 1)[0])
    
            for line in currentfile:
                out.write('\n')
                words = line.split()
                newwords = [str(float(words[0])+offsetx), str(float(words[1])+offsety), words[2], str((int(words[3]) + 2050) / 2)]              
                out.write(" ".join(newwords))
    

I also added a few Python best-practices, and you don't need to turn
`words[2]` into a float, then back to a string again.

You could also look into using the `csv` module, it can handle splitting and
rejoining lines in C code:

    
    
    import csv
    
    for file in files:
        with open(file, "rb") as currentfile, open(file[:-4]+"_RGB_moved.pts", "wb") as out:
            reader = csv.reader(currentfile, delimiter=' ', quoting=csv.QUOTE_NONE)
            writer = csv.writer(out, delimiter=' ', quoting=csv.QUOTE_NONE)
    
            out.writerow(next(reader)[0])
    
            for row in reader:
                newrow = [str(float(row[0])+offsetx), str(float(row[1])+offsety), row[2], str((int(row[3]) + 2050) / 2)]              
                out.writerow(newrow)
    

