title: read files with different encoding format using sys.stdin in python3
body: <p>I have many files which are encoded with UTF-8 or GBK. My system encoding is UTF-8 (<code>LANG=zh_CN.UTF-8</code>), so I can read files encoded with UTF-8 easily. But I must read file encoding with GBK as well. I'm following <a href="https://stackoverflow.com/questions/16549332/python-3-how-to-specify-stdin-encoding">Python 3: How to specify stdin encoding</a> here:</p>
  <pre><code>import sys 
  import io
  input_stream = io.TextIOWrapper(sys.stdin.buffer, encoding='gbk')
  for line in input_stream:
      print(line)
  </code></pre>
  <p>My question is how can I read all the files (both GBK and UTF-8) safely from  <code>sys.stdin</code>. Or can you give me some better solution?</p>
  <p>To slightly expand on this question, I want to handle files like this:</p>
  <pre><code>cat *.in | python3 handler.py
  </code></pre>
  <p><code>*.in</code> returns many files encoded with either UTF-8 or GBK.</p>
  <p>If I use the following code in <code>handler.py</code></p>
  <pre><code>for line in sys.stdin:
      ...some code
  </code></pre>
  <p>it will throw an error as soon as it tries to process a GBK file:</p>
  <pre><code>UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd5 in position 0: invalid continuation byte
  </code></pre>
  <p>On the other hand, if I use code like this:</p>
  <pre><code>input_stream = io.TextIOWrapper(sys.stdin.buffer, encoding='gbk')
  for line in input_stream:
      ...some code
  </code></pre>
  <p>it will throw an error on any UTF-8 file:</p>
  <pre><code>UnicodeDecodeError: 'gbk' codec can't decode byte 0x80 in position 25: illegal multibyte sequence
  </code></pre>
  <p>I want to find a safe way to handle both types of files (UTF-8 and GBK) within my script.</p>