title: Justadistraction: tokenizing English without whitespaces. Murakami SheepMan
body: <p>I wondered how <strong>you</strong> would go about tokenizing strings in English (or other western languages) if whitespaces were removed?</p>
  
  <p>The inspiration for the question is the Sheep Man character in the Murakami novel '<a href="http://en.wikipedia.org/wiki/Dance_Dance_Dance" rel="noreferrer">Dance Dance Dance</a>'</p>
  
  <p>In the novel, the Sheep Man is translated as saying things like:</p>
  
  <blockquote>
    <p>"likewesaid, we'lldowhatwecan. Trytoreconnectyou, towhatyouwant," said the Sheep Man. "Butwecan'tdoit-alone. Yougottaworktoo."</p>
  </blockquote>
  
  <p>So, some punctuation is kept, but not all. Enough for a human to read, but somewhat arbitrary.</p>
  
  <p>What would be your strategy for building a parser for this? Common combinations of letters, syllable counts, conditional grammars, look-ahead/behind regexps etc.?</p>
  
  <p>Specifically, python-wise, how would you structure a (forgiving) translation flow? Not asking for a completed answer, just more how your thought process would go about breaking the problem down.</p>
  
  <p>I ask this in a frivolous manner, but I think it's a question that might get some interesting (nlp/crypto/frequency/social) answers.
  Thanks!</p>