title: How do applications know character encoding?
body: <p>Lets say I have two files as below : </p>
  
  <pre><code>$ ll
  total 8
  -rw-rw-r--. 1 matias matias 6 Nov 27 20:25 ascii.txt
  -rw-rw-r--. 1 matias matias 8 Nov 28 21:57 unicode.txt
  </code></pre>
  
  <p>Both contain a single line of text, but there is an extra character in the second file as shown here ( Greek letter Sigma ) : </p>
  
  <pre><code>$ cat ascii.txt
     matias
  $ cat unicode.txt
     matiasΣ
  </code></pre>
  
  <p>If I pass them through <strong>file</strong> command this is the output :</p>
  
  <pre><code>$ file *
    ascii.txt:   ASCII text, with no line terminators
    unicode.txt: UTF-8 Unicode text, with no line terminators
  </code></pre>
  
  <p>Which seems ok. Now If I make an <strong>hexdump</strong> of the file I get this :</p>
  
  <pre><code> $ hexdump -C ascii.txt
     00000000  6d 61 74 69 61 73                     |matias|
     00000006
  
   $ hexdump -C unicode.txt
     00000000  6d 61 74 69 61 73 ce a3               |matias..|
     00000008
  </code></pre>
  
  <p>So, my question is, how does an application as <strong>cat</strong> know that the last two bytes are actually a <strong>single Unicode character</strong>. If I print the last two bytes individually I get:</p>
  
  <pre><code> $ printf '%d' '0xce'
     206
   $ printf '%d' '0xa3'
     163
  </code></pre>
  
  <p>Which in extended ASCII are :</p>
  
  <pre><code> $ py3 -c 'print(chr(206))'
     Î
   $ py3 -c 'print(chr(163))'
     £
  </code></pre>
  
  <p>Is my logic flawed? What Am I missing here?</p>