title: How to do a Python split() on languages (like Chinese) that don't use whitespace as word separator?
body: <p>I want to split a sentence into a list of words.</p>
  
  <p>For English and European languages this is easy, just use split()</p>
  
  <pre><code>&gt;&gt;&gt; "This is a sentence.".split()
  ['This', 'is', 'a', 'sentence.']
  </code></pre>
  
  <p>But I also need to deal with sentences in languages such as Chinese that don't use whitespace as word separator.</p>
  
  <pre><code>&gt;&gt;&gt; u"这是一个句子".split()
  [u'\u8fd9\u662f\u4e00\u4e2a\u53e5\u5b50']
  </code></pre>
  
  <p>Obviously that doesn't work.</p>
  
  <p>How do I split such a sentence into a list of words?</p>
  
  <p><strong>UPDATE:</strong></p>
  
  <p>So far the answers seem to suggest that this requires natural language processing techniques and that the word boundaries in Chinese are ambiguous. I'm not sure I understand why. The word boundaries in Chinese seem very definite to me. Each Chinese word/character has a corresponding unicode and is displayed on screen as an separate word/character.</p>
  
  <p>So where does the ambiguity come from. As you can see in my Python console output Python has no problem telling that my example sentence is made up of 5 characters:</p>
  
  <pre><code>这 - u8fd9
  是 - u662f
  一 - u4e00
  个 - u4e2a
  句 - u53e5
  子 - u5b50
  </code></pre>
  
  <p>So obviously Python has no problem telling the word/character boundaries. I just need those words/characters in a list.</p>