Python - Output creating blank spaces
<p>I'm encountering a very strange issue in a python script I've written. This is the piece of code that is producing abnormal results:</p>

<p><strong>EDIT: I've included the entire loop in the code segment now.</strong></p>

<pre><code>data = open(datafile,'r')
outERROR = open(outERRORfile,'w')
precision=[0]
scale=[0]
lines = data.readlines()
limit = 0
if filetype == 'd':
    for line in lines:
            limit += 1
            if limit &gt; checklimit:
                    break
            columns = line.split(fieldDelimiter)
            for i in range(len(columns) - len(precision)):
                            precision.append(0)
            for i in range(len(columns) - len(scale)):
                            scale.append(0)
            if len(datatype) != len(precision):
                    sys.exit() #Exits the script if the number of data types (fields found in the DDL file) doesn't match the number of columns found in the data file
            i = -1
            for eachcolumn in columns:
                    i += 1
                    if len(rstrip(columns[i])) &gt; precision[i]:
                            precision[i] = len(rstrip(columns[i]))
                    if columns[i].find('.') != -1 and (len(rstrip(columns[i])) - rstrip(columns[i]).find('.')) &gt; scale[i]:
                            scale[i] = len(rstrip(columns[i])) - rstrip(columns[i]).find('.') -1
                    if datatype[i][0:7] == 'integer':
                            if int(columns[i]) &lt; -2147483648 or int(columns[i]) &gt; 2147483647:
                                    outERROR.write("Integer value too high or too low to fit inside Integer data type, column: " + str(i + 1) + ", value: " + columns[i] + "\n")
                    if datatype[i][0:9] == 'smallint':
                            if int(columns[i]) &lt; -32768 or int(columns[i]) &gt; 32767:
                                    outERROR.write("Smallint value too high or too low to fit inside Smallint data type, column: " + str(i + 1) + ", value: " + columns[i] + "\n")
                    if datatype[i][0:7] == 'byteint':
                            if int(columns[i]) &lt; -128 or int(columns[i]) &gt; 127:
                                    outERROR.write("Byteint value too high or too low to fit inside Byteint data type, column: " + str(i + 1) + ", value: " + columns[i] + "\n")
                    if datatype[i][0:4] == 'date':
                            if DateParse(columns[i],format1[i]) &gt; -1:
                                    pass
                            elif DateParse(columns[i],format2[i]) &gt; -1:
                                    pass
                            elif DateParse(columns[i],format3[i]) &gt; -1:
                                    pass
                            else:
                                    outERROR.write('Date format error, column: ' + str(i + 1) + ', value: ' + columns[i])
                    if datatype[i][0:9] == 'timestamp':
                            if DateParse(columns[i],timestamp1[i]) &gt; -1:
                                    pass
                            elif DateParse(columns[i],timestamp2[i]) &gt; -1:
                                    pass
                            elif DateParse(columns[i],timestamp3[i]) &gt; -1:
                                    pass
                            else:
                                    outERROR.write('Timestamp format error, column: ' + str(i + 1) + ', value: ' + columns[i] + '\n')
                    if (datatype[i][0:7] == 'decimal'
                    or datatype[i][0:7] == 'integer'
                    or datatype[i][0:7] == 'byteint'
                    or datatype[i][0:5] == 'float'
                    or datatype[i][0:8] == 'smallint'):
                            try:
                                    y = float(columns[i])
                            except ValueError:
                                    outERROR.write('Character found in numeric data type, column: ' + str(i + 1) + ', value: ' + columns[i] + "\n")
                    else:
                            pass
</code></pre>

<p>This is part of a loop that reads a data file, basically its checking the type of the data (to determine if its supposed to be a numeric type) and 'trying' to turn it into a float to see if its actually numeric data in the data file. If its not numeric data it outputs the error you see above to a text file (defined currently as outERROR). Now when I wrote this and tested it on a small data file (4 lines) it worked fine, but when I run this on a larger file (several thousand rows) my error file is suddenly filling with a bunch of blank spaces, and only a few of the error messages are being created.</p>

<p>Here is what the error file looks like when I run the script with 4 rows:</p>

<pre><code>Character found in numeric data type, column: 6, value: 24710a35
Character found in numeric data type, column: 7, value: 0a04
Character found in numeric data type, column: 8, value: 0a02
Character found in numeric data type, column: 6, value: 56688a12
Character found in numeric data type, column: 7, value: 0a09
Character found in numeric data type, column: 8, value: 0a06
Character found in numeric data type, column: 6, value: 12301a04
Character found in numeric data type, column: 7, value: 0a10
Character found in numeric data type, column: 8, value: 0a02
Character found in numeric data type, column: 6, value: 25816a56
Character found in numeric data type, column: 7, value: 0a09
Character found in numeric data type, column: 8, value: 0a06
</code></pre>

<p>This is the expected output.</p>

<p>When I run it on larger files, I start to get blank spaces at the top of the error file, and only the last 40-50 or so error writes actually get output as text in the file. The larger the file, the more blank spaces it outputs. I'm completely lost on this, I've read some of the other questions regarding mysterious blank lines and spaces on stackoverflow.com here but they dont seem to address my issue.</p>

<p><strong>EDIT: outERROR is the name I've given to the error file that the output is writing to. It is a simple .txt file.</strong></p>

<p>This is a sample of the data file:</p>

<pre><code>257|1463|64|1|7|9551a22|0a05|0a02|N|O|1998-06-18|1998-05-15|1998-06-27|COLLECT COD|FOB|ackages sleep bold realmsa f|
258|1062|68|1|8|7704a48|0a00|0a07|R|F|1994-01-20|1994-03-21|1994-02-09|NONE|REG AIR|ully about the fluffily silent dependencies|
258|1962|95|2|40|74558a40|0a10|0a01|A|F|1994-03-13|1994-02-23|1994-04-05|DELIVER IN PERSON|FOB|silent frets nod daringly busy, bold|
258|1618|19|3|45|68382a45|0a07|0a07|R|F|1994-03-04|1994-02-13|1994-03-30|DELIVER IN PERSON|TRUCK|regular excuses-- fluffily ruthl|
</code></pre>

<p>Specifically the columns that are causing output to the error file are:</p>

<pre><code>|9551a22|0a05|0a02|
|7704a48|0a00|0a07|
|74558a40|0a10|0a01|
|68382a45|0a07|0a07|
</code></pre>

<p>So each line should cause 3 writes to the error file, specifying these values. It works fine for a small number of lines, but when it reads a large number of lines I start getting these mysterious blank spaces. This problem occurs <strong>only</strong> when I have numeric fields that contain characters.</p>
