title: Ruby's String#gsub, unicode, and non-word characters
body: <p>As part of a larger series of operations, I'm trying to take tokenized chunks of a larger string and get rid of punctuation, non-word gobbledygook, etc. My initial attempt used <code>String#gsub</code> and the <code>\W</code> regexp character class, like so:</p>
  
  <pre><code>my_str = "Hello,"
  processed = my_str.gsub(/\W/,'')
  puts processed # =&gt; Hello
  </code></pre>
  
  <p>Super, super, super simple. Of course, now I'm extending my program to deal with non-Latin characters, and all heck's broken loose. Ruby's <code>\W</code> seems to be something like <code>[^A-Za-z0-9_]</code>, which, of course, excludes stuff with diacritics (ü, í, etc.). So, now my formerly-simple code crashes and burns in unpleasent ways:</p>
  
  <pre><code>my_str = "Quística."
  processed = my_str.gsub(/\W/,'')
  puts processed # =&gt; Qustica
  </code></pre>
  
  <p>Notice that gsub() obligingly removed the accented "í" character. One way I've thought of to fix this would be to extend Ruby's \W whitelist to include higher Unicode code points, but there are an awful lot of them, and I know I'd miss some and cause problems down the line (and let's not even start thinking about non-Latin languages...). Another solution would be to blacklist all the stuff I want to get rid of (punctuation, $/%/&amp;/™, etc.), but, again, there's an awful lot of that and I <em>really</em> don't want to start playing blacklist-whack-a-mole.</p>
  
  <p>Has anybody out there found a principled solution to this problem? Is there some hidden, Unicode-friendly version of <code>\W</code> that I haven't discovered yet? Thanks!</p>