title: Why does maven give me different utf-8 characters than eclipse (test run in eclipse, fail in maven)?
body: <p>My current project is concerned with parsing natural language. One test reads text from a file, removes certain characters, and tokenizes the text into single words. The test actually compares the number of unique words. In eclipse, this test is "green", in maven, I get a higher number of words than expected. Comparing the lists of words, I see the following additional words:</p>
  
  <ul>
  <li>acquirer⊙s</li>
  <li>card⊙s</li>
  <li>institution⊙s</li>
  <li>issuer⊙s</li>
  <li>provider⊙s</li>
  <li>psam⊙s</li>
  <li>⊜from⊝</li>
  <li>⊜slot⊝</li>
  <li>⊜to⊝</li>
  </ul>
  
  <p>Looking at the text source, it contains the following characters which should be filtered away: “ ” ’</p>
  
  <p>This works in eclipse, but not in maven. I am using utf-8. The files seem to be encoded correctly, in the maven pom I specify the following:</p>
  
  <pre><code>&lt;properties&gt;
          &lt;project.build.sourceEncoding&gt;UTF-8&lt;/project.build.sourceEncoding&gt;
          &lt;org.apache.lucene.version&gt;3.6.0&lt;/org.apache.lucene.version&gt;
  &lt;/properties&gt;
  </code></pre>
  
  <p>Edit: Here is the code that reads the file (which is, according to eclipse, encoded as UTF-8).</p>
  
  <pre><code>        BufferedReader reader = new BufferedReader(
                  new FileReader(this.file));
          String line = "";
          while ((line = reader.readLine()) != null) {
              // the csv contains a text and a classification
              String[] reqCatType = line.split(";");
              String reqText = reqCatType[0].trim();
              String reqCategory = reqCatType[1].trim();
              // the tokenizer also removes unwanted characters:
              String[] sentence = this.filter.filterStopWords(this.tokenizer
                      .tokenize(reqText));
              // we use this data to train a machine learning algorithm
              this.dataSet.learn(sentence, reqCategory);
          }
          reader.close();
  </code></pre>
  
  <p>Edit: The following information might be useful for analyzing the problem:</p>
  
  <pre><code>mvn -v
  Apache Maven 3.0.3 (r1075438; 2011-02-28 09:31:09-0800)
  Maven home: /usr/share/maven
  Java version: 1.6.0_33, vendor: Apple Inc.
  Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
  Default locale: en_US, platform encoding: MacRoman
  OS name: "mac os x", version: "10.6.8", arch: "x86_64", family: "mac"
  </code></pre>