title: How to set whitespace tokenizer on NER Model?
body: <p>i am creating a custom NER model using CoreNLP 3.6.0</p>
  
  <p>My props are:</p>
  
  <pre><code># location of the training file 
  trainFile = /home/damiano/stanford-ner.tsv 
  # location where you would like to save (serialize) your 
  # classifier; adding .gz at the end automatically gzips the file, 
  # making it smaller, and faster to load 
  serializeTo = ner-model.ser.gz
  
  # structure of your training file; this tells the classifier that 
  # the word is in column 0 and the correct answer is in column 1 
  map = word=0,answer=1
  
  # This specifies the order of the CRF: order 1 means that features 
  # apply at most to a class pair of previous class and current class 
  # or current class and next class. 
  maxLeft=1
  
  # these are the features we'd like to train with 
  # some are discussed below, the rest can be 
  # understood by looking at NERFeatureFactory 
  useClassFeature=true 
  useWord=true 
  # word character ngrams will be included up to length 6 as prefixes 
  # and suffixes only  
  useNGrams=true 
  noMidNGrams=true 
  maxNGramLeng=6 
  usePrev=true 
  useNext=true 
  useDisjunctive=true 
  useSequences=true 
  usePrevSequences=true 
  # the last 4 properties deal with word shape features 
  useTypeSeqs=true 
  useTypeSeqs2=true 
  useTypeySequences=true 
  wordShape=chris2useLC
  </code></pre>
  
  <p>I build with this command:</p>
  
  <pre><code>java -classpath "stanford-ner.jar:lib/*" edu.stanford.nlp.ie.crf.CRFClassifier  -prop /home/damiano/stanford-ner.prop
  </code></pre>
  
  <p>The problem is when i use this model to retrieve the entities inside a textfile. The command is:</p>
  
  <pre><code>java -classpath "stanford-ner.jar:lib/*" edu.stanford.nlp.ie.crf.CRFClassifier -loadClassifier ner-model.ser.gz -textFile file.txt
  </code></pre>
  
  <p>Where <strong>file.txt</strong> is:</p>
  
  <pre><code>Hello!
  my
  name
  is
  John.
  </code></pre>
  
  <p>The output is:</p>
  
  <p>Hello/O !/O 
  my/O name/O is/O John/PERSON ./O </p>
  
  <p>As you can see it split "Hello!" into two tokens. Same thing for "John."</p>
  
  <p>I must use whitespace tokenizer. </p>
  
  <p>How can i set it? </p>
  
  <p>why does CoreNlp is splitting those words in two tokens?</p>