title: What does ($pre =~ /\./ && $pre =~ /\p{IsAlpha}/) mean in the Moses Tokenizer?
body: <p><a href="https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl#L333" rel="nofollow noreferrer">Moses Tokenizer</a> is the tokenizer widely used in machine translation and natural language processing experiments.</p>
  
  <p>There is a line of regex that checks for:</p>
  
  <pre><code>if (($pre =~ /\./ &amp;&amp; $pre =~ /\p{IsAlpha}/) || 
     ($NONBREAKING_PREFIX{$pre} &amp;&amp; $NONBREAKING_PREFIX{$pre}==1) || 
     ($i&lt;scalar(@words)-1 &amp;&amp; ($words[$i+1] =~ /^[\p{IsLower}]/)))
  </code></pre>
  
  <p>Please correct me if I'm wrong, the 2nd and 3rd conditions are to check</p>
  
  <ul>
  <li>whether the prefix is in a list of nonbreaking prefixes </li>
  <li>whether the word is not the last token and there is still a lowercased token as the next word.</li>
  </ul>
  
  <p>The question is on the first condition where it checks for:</p>
  
  <pre><code>($pre =~ /\./ &amp;&amp; $pre =~ /\p{IsAlpha}/)
  </code></pre>
  
  <ol>
  <li><p>Is the <code>$pre =~ /\./</code> checking whether the prefix is a single fullstop?</p></li>
  <li><p>And is <code>$pre =~ /\p{IsAlpha}/</code> checking whether the prefix is an alpha from the list of alphabet in the <a href="http://perldoc.perl.org/perluniprops.html" rel="nofollow noreferrer">perluniprop</a>?</p></li>
  <li><p>One related question is whether the fullstop is already inside the perluniprop alphabet? If so, wouldn't this condition never be true?</p></li>
  </ol>