java StreamTokenizer wordChars() and nextToken()
<p>This might be a dumb question but I am having a hard time recognizing how StreamTokenizer delimit input streams. Is it delimited by space and nextline? I am also confused with the use of wordChars(). For example:</p>

<pre><code>public static int getSet(String workingDirectory, String filename, List&lt;String&gt; set) {
    int cardinality = 0;
    File file = new File(workingDirectory,filename);
    try {
        BufferedReader in = new BufferedReader(new FileReader(file));
        StreamTokenizer text = new StreamTokenizer(in);
        text.wordChars('_','_');
        text.nextToken();
        while (text.ttype != StreamTokenizer.TT_EOF) {
            set.add(text.sval);
            cardinality++;
//              System.out.println(cardinality + " " + text.sval);
            text.nextToken();
        }
        in.close();
    } catch (IOException ex) {
        ex.printStackTrace();
    }
    return cardinality; 
}
</code></pre>

<p>If the text file includes such string:A_B_C D_E_F.</p>

<p>Does text.wordChars('_','_') mean only underscore will be considered as valid words?</p>

<p>And what will the tokens be in this case?</p>

<p>Thank you very much.</p>
