title: UnicodeDecodeError: unexpected end of data
body: <p>I have a huge text file which I want to open.<br>
  I'm reading the file in chunks, avoiding memory issues related to reading too much of the file all at once.  </p>
  
  <p><strong>code snippet:</strong></p>
  
  <pre><code>def open_delimited(fileName, args):
  
      with open(fileName, args, encoding="UTF16") as infile:
          chunksize = 10000
          remainder = ''
          for chunk in iter(lambda: infile.read(chunksize), ''):
              pieces = re.findall(r"(\d+)\s+(\d+_\d+)", remainder + chunk)
              for piece in pieces[:-1]:
                  yield piece
              remainder = '{} {} '.format(*pieces[-1]) 
          if remainder:
              yield remainder
  </code></pre>
  
  <p>the code throws the error <code>UnicodeDecodeError: 'utf16' codec can't decode bytes in position 8190-8191: unexpected end of data</code>.</p>
  
  <p>I tried <code>UTF8</code> and got the error <code>UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 0: invalid start byte</code>.  </p>
  
  <p><code>latin-1</code> and <code>iso-8859-1</code> raised the error <code>IndexError: list index out of range</code> </p>
  
  <p>A sample of the input file:</p>
  
  <pre><code>b'\xff\xfe1\x000\x000\x005\x009\x00\t\x001\x000\x000\x005\x009\x00_\x009\x007\x004\x007\x001\x007\x005\x003\x001\x000\x009\x001\x00\t\x00\t\x00P\x00o\x00s\x00t\x00\t\x001\x00\t\x00H\x00a\x00p\x00p\x00y\x00 \x00B\x00i\x00r\x00t\x00h\x00d\x00a\x00y\x00\t\x002\x000\x001\x001\x00-\x000\x008\x00-\x002\x004\x00 \x00'
  </code></pre>
  
  <p>I will also mention that I have several of those huge text files.<br>
  <code>UTF16</code> works fine for many of them, and fail at a specific file.</p>
  
  <p>Anyway to resolve this issue?</p>