title: wiki dump encoding
body: <p>I'm using WikiPrep to process the latest wiki dump enwiki-20121101-pages-articles.xml.bz2.  Instead of "use Parse::MediaWikiDump;" I replaced that by "use MediaWiki::DumpFile::Compat;" and did the proper changes in the code. Then, I ran </p>
  
  <pre><code>perl wikiprep.pl -f enwiki-20121101-pages-articles.xml.bz2
  </code></pre>
  
  <p>I got an error</p>
  
  <pre><code>enwiki-20121101-pages-articles.xml.bz2:1: parser error : Document is empty
  BZh91AY&amp;SY±H¦ÂOÿ~Ð`ÿÿÿ¿ÿÿÿ¿ÿÿÿÿÿÿÿÿÿÿ½ÿýþdß8õEnÞ¶zëJ¨Eà®mEÓP|f÷Ô
  ^
  </code></pre>
  
  <p>I guess there are some non-utf8 characters contained in the dump. So I ran</p>
  
  <pre><code>iconv -f utf8 -t utf8 enwiki-20121101-pages-articles.xml.bz2
  </code></pre>
  
  <p>And indeed, I got some errors</p>
  
  <pre><code>BZh91AY&amp;SYiconv: illegal input sequence at position 10
  </code></pre>
  
  <p>So, my question is what's the encoding format of wiki dump and if I wish to convert it to utf-8, what shall I do? Or how should modify wikiprep.pl to avoid such problems.</p>
  
  <p>Many thanks</p>
  
  <p>-- [solved] I should first unzip the file first. </p>