title: Elasticsearch: Special characters with ICU plugin
body: <p>So I used this "tutorial" to start working with special characters in elasticsearch: <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/case-folding.html" rel="nofollow">https://www.elastic.co/guide/en/elasticsearch/guide/current/case-folding.html</a></p>
  
  <p>I installed the elasticsearch-analysis-icu version 2.7.0 for my elasticsearch version 1.7.x</p>
  
  <p>Creating an index "sonderzeichen" with the icu_tokenizer worked fine (I use nodejs):</p>
  
  <pre><code>    var http = require('http');
      var body = JSON.stringify(
              {
                  "settings": {
                      "analysis": {
                          "analyzer": {
                              "my_lowercaser": {
                                  "tokenizer": "icu_tokenizer",
                                  "filter":  [ "icu_normalizer" ] 
                              }
                          }
                      }
                  }
              }
      );
  
      var options = {
              host: 'localhost',
              path: '/sonderzeichen',
              port: 9200,
              method: "PUT",
              headers: {
                  'Content-Type': 'application/json',
                  'Content-Length': body.length
              }
      };
  
      callback = function(response) {
          var str = '';
          response.on('data', function(chunk){
              str += chunk;
          });
  
          response.on('end', function(){
              console.log(str);
          });
      };
  
  
      http.request(options, callback).end(body);
  </code></pre>
  
  <p>I used the two analyzers as described in the tutorial:</p>
  
  <pre><code>/_analyze?analyzer=my_lowercaser
  </code></pre>
  
  <p>and</p>
  
  <pre><code>/sonderzeichen/_analyze?analyzer=my_lowercaser
  </code></pre>
  
  <p>In node it looks like this:</p>
  
  <pre><code>    var http = require('http');
  
      var body = decodeURIComponent("Weißkopfseeadler WEISSKOPFSEEADLER äÄöÖüÜßáÁéÉíÍóÓúÚàÀèÈìÌòÒùÙ");
  
      var options = {
              host: 'localhost',
              path: '/_analyze?analyzer=standard',
              port: 9200,
              method: "GET",
              headers: {
                  'Content-Type': 'application/json',
                  'Content-Length': body.length
              }
      };
  
      callback = function(response) {
          var str = '';
          response.on('data', function(chunk){
              str += chunk;
          });
  
          response.on('end', function(){
              console.log(str);
          });
      };
  
  
      http.request(options, callback).end(body);
  </code></pre>
  
  <p>Both return the exact same token, as follows (no matter if I use decodeURIComponent or not):</p>
  
  <pre><code>    {
        "tokens": [
          {
            "token": "wei",
            "start_offset": 0,
            "end_offset": 3,
            "type": "&lt;ALPHANUM&gt;",
            "position": 1
          },
          {
            "token": "kopfseeadler",
            "start_offset": 4,
            "end_offset": 16,
            "type": "&lt;ALPHANUM&gt;",
            "position": 2
          },
          {
            "token": "weisskopfseeadler",
            "start_offset": 17,
            "end_offset": 34,
            "type": "&lt;ALPHANUM&gt;",
            "position": 3
          }
        ]
      }
  </code></pre>
  
  <p>Elastic still can't seem to handle any special characters, so where did I go wrong?</p>