title: Tokenize Arabic text files java
body: <p>I am trying to tokenize some text files into words and I write this code, It works perfect in English and when I try it in Arabic it did not work.
  I added the UTF-8 to read Arabic files. did I miss something</p>
  
  <pre><code>public void parseFiles(String filePath) throws FileNotFoundException, IOException {
      File[] allfiles = new File(filePath).listFiles();
      BufferedReader in = null;
      for (File f : allfiles) {
          if (f.getName().endsWith(".txt")) {
              fileNameList.add(f.getName());
              Reader fstream = new InputStreamReader(new FileInputStream(f),"UTF-8"); 
             // BufferedReader br = new BufferedReader(fstream);
              in = new BufferedReader(fstream);
              StringBuilder sb = new StringBuilder();
              String s=null;
              String word = null;
              while ((s = in.readLine()) != null) {
                  Scanner input = new Scanner(s);
                    while(input.hasNext()) {
                         word = input.next();
                  if(stopword.isStopword(word)==true)
                  {
                      word= word.replace(word, "");
                  }
  
                  //String stemmed=stem.stem (word);
                  sb.append(word+"\t");
                    }
                     //System.out.print(sb);  ///here the arabic text is outputed without stopwords
  
  
              }
              String[] tokenizedTerms = sb.toString().replaceAll("[\\W&amp;&amp;[^\\s]]", "").split("\\W+");   //to get individual terms
  
              for (String term : tokenizedTerms) {
                  if (!allTerms.contains(term)) {  //avoid duplicate entry
                      allTerms.add(term);
                      System.out.print(term+"\t");  //here the problem.
                  }
              }
              termsDocsArray.add(tokenizedTerms);
          }
      }
  
  } 
  </code></pre>
  
  <p>Please any ideas to help me proceed.
  Thanks</p>