I am using the Lazy Method for reading big files to get extract information from text files to store it on a database.

```
def read_in_chunks(file_object, chunk_size=1024):
"""Lazy function (generator) to read a file piece by piece.
Default chunk size: 1k."""
while True:
    data = file_object.read(chunk_size)
    if not data:
        break
    yield data

f = open('really_big_file.dat')
for piece in read_in_chunks(f):
    process_data(piece) 

```

The problem is that if I modify the process\_data(piece) to for example:

```
for data in piece:
    data = regex.findall(str(data)) 
    cursor.execute('INSERT INTO table (Title) VALUES (?)', data)

```

I will get a blank space every time a new chunk is being read. In this case it is trivial to modify it to:

```
   for data in piece:
        data = regex.findall(str(data)) 
        if (data != ""):
             cursor.execute('INSERT INTO table (Title) VALUES (?)', data[1])

```

But in the information I am extracting there is some valid "blank spaces" (when there is not certain information in some fields. How can I avoid this blank space between chunks??? I know this is not a problem with the way I am extracting the information because if I extract smaller files without using the lazy method I don't have this problem. But I have certain files that are much bigger and it works really slow without the lazy method.

Thanks