title: java.util.Scanner and Wikipedia
body: <p>I'm trying to use java.util.Scanner to take Wikipedia contents and use it for word based searches. 
  The fact is that it's all fine but when reading some words it give me errors.
  Looking at code and making some check it turned out that with some words it seems 
  not to recognize the encoding, or so, and the content is no more readable.
  This is the code used to take the page:</p>
  
  <p>// -Start-</p>
  
  <pre><code>try {
          connection =  new URL("http://it.wikipedia.org
  wiki/"+word).openConnection();
                      Scanner scanner = new Scanner(connection.getInputStream());
          scanner.useDelimiter("\\Z");
          content = scanner.next();
  //          if(word.equals("pubblico"))
  //              System.out.println(content);
          System.out.println("Doing: "+ word);
  //End
  </code></pre>
  
  <p>The problem arises with words as "pubblico" for the italian wikipedia.
  the result of the println on word pubblico is like this (cutted):
  ï¿ï¿½]Ksr>ï¿½~E
  ï¿½1Aï¿½ï¿½ï¿½Eï¿½ER3tHZï¿½4vï¿½ï¿½&amp;PZjtcï¿½¿½ï¿½Dï¿½7_|ï¿½ï¿½ï¿½ï¿½=8ï¿½ï¿½Ø}</p>
  
  <p>Do you have any idea why? Yet looked at page source and headers are the same, with same encoding...</p>
  
  <p>Turned Out that content is gzipped, so can i tell wikipedia not to send me teir pages zipped or it's the only way? thank you</p>