StringTokenizer is used to tokenize a tagged string in JAVA. The string is tagged using Parts Of Speech MaxentTagger of Stanford. Substring of the tagged text is taken to display just the POS tag and just the word iteratively.

Here's the text before tagging:

> 
> Man has always had this notion that brave deeds are manifest in physical actions. While it is not entirely erroneous, there doesn't lie the singular path to valor. From of old, it is a sign of strength to fight back a wild animal. It is understandable if fought in defense; however, to go the extra mile and instigate an animal and fight it is the lowest degree of civilization man can exhibit. More so, in this age of reasoning and knowledge. Tradition may call it, but adhering blindly to it is idiocy, be it the famed Jallikattu in Tamil Nadu (The Indian equivalent to the Spanish Bullfighting) or the cock-fights. Pelting stones at a dog and relishing it howl in pain is dreadful. If one only gave as much as a trickle of thought and conscience the issue would surface as deplorable in every aspect. Animals play a part along with us in our ecosystem. And, some animals are dearer: the stray dogs that guard our street, the intelligent crow, the beast of burden and the everyday animals of pasture. Literature has voiced in its own way: In The Lord of the Rings the fellowship treated Bill Ferny's pony with utmost care; in Harry Potter when they didn’t heed Hermione's advice on the treatment of house elves they learned the hard way that it caused their own undoing; and Jack London, writes all about animals.Indeed, Kindness to animals is a virtue. 
> 
> 
> 

Here's the POS tagged text:

> 
> Man\_NN has\_VBZ always\_RB had\_VBN this\_DT notion\_NN that\_IN brave\_VBP deeds\_NNS are\_VBP manifest\_JJ in\_IN physical\_JJ actions\_NNS .\_. While\_IN it\_PRP is\_VBZ not\_RB entirely\_RB erroneous\_JJ ,\_, there\_EX does\_VBZ n't\_RB lie\_VB the\_DT singular\_JJ path\_NN to\_TO valor\_NN .\_. From\_IN of\_IN old\_JJ ,\_, it\_PRP is\_VBZ a\_DT sign\_NN of\_IN strength\_NN to\_TO fight\_VB back\_RP a\_DT wild\_JJ animal\_NN .\_. It\_PRP is\_VBZ understandable\_JJ if\_IN fought\_VBN in\_IN defense\_NN ;\_: however\_RB ,\_, to\_TO go\_VB the\_DT extra\_JJ mile\_NN and\_CC instigate\_VB an\_DT animal\_NN and\_CC fight\_VB it\_PRP is\_VBZ the\_DT lowest\_JJS degree\_NN of\_IN civilization\_NN man\_NN can\_MD exhibit\_VB .\_. More\_RBR so\_RB ,\_, in\_IN this\_DT age\_NN of\_IN reasoning\_NN and\_CC knowledge\_NN .\_. Tradition\_NN may\_MD call\_VB it\_PRP ,\_, but\_CC adhering\_JJ blindly\_RB to\_TO it\_PRP is\_VBZ idiocy\_NN ,\_, be\_VB it\_PRP the\_DT famed\_JJ Jallikattu\_NNP in\_IN Tamil\_NNP Nadu\_NNP -LRB-\_-LRB- The\_DT Indian\_JJ equivalent\_NN to\_TO the\_DT Spanish\_JJ Bullfighting\_NN -RRB-\_-RRB- or\_CC the\_DT cock-fights\_NNS .\_. Pelting\_VBG stones\_NNS at\_IN a\_DT dog\_NN and\_CC relishing\_VBG it\_PRP howl\_NN in\_IN pain\_NN is\_VBZ dreadful\_JJ .\_. If\_IN one\_CD only\_RB gave\_VBD as\_RB much\_JJ as\_IN a\_DT trickle\_VB of\_IN thought\_NN and\_CC conscience\_NN the\_DT issue\_NN would\_MD surface\_VB as\_IN deplorable\_JJ in\_IN every\_DT aspect\_NN .\_. Animals\_NNS play\_VBP a\_DT part\_NN along\_IN with\_IN us\_PRP in\_IN our\_PRP$ ecosystem\_NN .\_. And\_CC ,\_, some\_DT animals\_NNS are\_VBP dearer\_RBR :\_: the\_DT stray\_JJ dogs\_NNS that\_WDT guard\_VBP our\_PRP$ street\_NN ,\_, the\_DT intelligent\_JJ crow\_NN ,\_, the\_DT beast\_NN of\_IN burden\_NN and\_CC the\_DT everyday\_JJ animals\_NNS of\_IN pasture\_NN .\_. Literature\_NN has\_VBZ voiced\_VBN in\_IN its\_PRP$ own\_JJ way\_NN :\_: In\_IN The\_DT Lord\_NN of\_IN the\_DT Rings\_NNP the\_DT fellowship\_NN treated\_VBN Bill\_NNP Ferny\_NNP 's\_POS pony\_NN with\_IN utmost\_JJ care\_NN ;\_: in\_IN Harry\_NNP Potter\_NNP when\_WRB they\_PRP did\_VBD n't\_RB heed\_VB Hermione\_NNP 's\_POS advice\_NN on\_IN the\_DT treatment\_NN of\_IN house\_NN elves\_NNS they\_PRP learned\_VBD the\_DT hard\_JJ way\_NN that\_IN it\_PRP caused\_VBD their\_PRP$ own\_JJ undoing\_NN ;\_: and\_CC Jack\_NNP London\_NNP ,\_, writes\_VBZ all\_DT about\_IN animals\_NNS .\_. Indeed\_RB ,\_, Kindness\_NN to\_TO animals\_NNS is\_VBZ a\_DT virtue\_NN .\_. 
> 
> 
> 

And here's the code which seeks to obtain the above mentioned substrings:

```
String line;
StringBuilder sb=new StringBuilder();
try(FileInputStream input = new FileInputStream("E:\\D.txt"))
    {
    int data = input.read();
    while(data != -1)
        {
        sb.append((char)data);
        data = input.read();
        }
    }
catch(FileNotFoundException e)
{
    System.err.println("File Not Found Exception : " + e.getMessage());
}
line=sb.toString();
String line1=line;//Copy for Tagger
line+=" T";       
List<String> sentenceList = new ArrayList<String>();//TAGGED DOCUMENT
MaxentTagger tagger = new MaxentTagger("E:\\Installations\\Java\\Tagger\\english-left3words-distsim.tagger");
String tagged = tagger.tagString(line1);
File file = new File("A.txt");
BufferedWriter output = new BufferedWriter(new FileWriter(file));
output.write(tagged);
output.close();
DocumentPreprocessor dp = new DocumentPreprocessor("C:\\Users\\Admin\\workspace\\Project\\A.txt");
int largest=50;
int m=0;
StringTokenizer st1;
for (List<HasWord> sentence : dp) 
{
   String sentenceString = Sentence.listToString(sentence);
   sentenceList.add(sentenceString.toString());
}
String[][] Gloss=new String[sentenceList.size()][largest];
String[] Adj=new String[largest];
String[] Adv=new String[largest];
String[] Noun=new String[largest];
String[] Verb=new String[largest];
int adj=0,adv=0,noun=0,verb=0;
for(int i=0;i<sentenceList.size();i++)
{
    st1= new StringTokenizer(sentenceList.get(i)," ,(){}[]/.;:&?!");
    m=0;//Count for Gloss 2nd dimension
    //GETTING THE POS's COMPARTMENTALISED
    while(st1.hasMoreTokens())
    {
        String token=st1.nextToken();
        if(token.length()>1)//TO SKIP PAST TOKENS FOR PUNCTUATION MARKS
        {
        System.out.println(token);
        String s=token.substring(token.lastIndexOf("_")+1,token.length());
        System.out.println(s);
        if(s.equals("JJ")||s.equals("JJR")||s.equals("JJS"))
        {
            Adj[adj]=token.substring(0,token.lastIndexOf("_"));
            System.out.println(Adj[adj]);
            adj++;
        }
        if(s.equals("NN")||s.equals("NNS"))
        {
            Noun[noun]=token.substring(0,  token.lastIndexOf("_"));
            System.out.println(Noun[noun]);
            noun++;
        }
        if(s.equals("RB")||s.equals("RBR")||s.equals("RBS"))
        {
            Adv[adv]=token.substring(0,token.lastIndexOf("_"));
            System.out.println(Adv[adv]);
            adv++;
        }
        if(s.equals("VB")||s.equals("VBD")||s.equals("VBG")||s.equals("VBN")||s.equals("VBP")||s.equals("VBZ"))
        {
            Verb[verb]=token.substring(0,token.lastIndexOf("_"));
            System.out.println(Verb[verb]);
            verb++;
        }
        }
    }
    i++;//TO SKIP PAST THE LINES WHERE AN EXTRA UNDERSCORE OCCURS FOR FULLSTOP
 }

```

D.txt contains the plain text. 

As for the issue:

Every word gets tokenized at the spaces. Except for 'n't\_RB' where it is tokenized as n't and RB separately. 

This is how the output looks:

```
Man_NN
NN
Man
has_VBZ 
VBZ
has
always_RB
RB
always
had_VBN
VBN
had
this_DT
DT
notion_NN
NN
notion
that_IN
IN
brave_VBP
VBP
brave
deeds_NNS
NNS
deeds
are_VBP
VBP
are
manifest_JJ
JJ
manifest
in_IN
IN
physical_JJ
JJ
physical
actions_NNS
NNS
actions
While_IN
IN
it_PRP
PRP
is_VBZ
VBZ
is
not_RB
RB
not
entirely_RB
RB
entirely
erroneous_JJ
JJ
erroneous
there_EX
EX
does_VBZ
VBZ
does
n't
n't
RB
RB

```

But if I just run 'there\_EX does\_VBZ n't\_RB lie\_VB' in the tokenizer 'n't\_RB' gets toknized together. When I run the program I get a StringIndexOutOfBounds Exception which is understandable because there's no '\_' in 'n't' or 'RB'.
Can anybody look to it? Thank you.