title: Pig : parsing line with blank delimiter
body: <p>I'm using Hadoop Pig (0.10.0) to process logs file, a log line looking like : </p>
  
  <pre><code>2012-08-01  INFO   (User:irim)   getListedStocksByMarkets completed in 7041 ms
  </code></pre>
  
  <p>I would like to get a relation with tokens split by blanks, that is :</p>
  
  <p><code>(2012-08-01,INFO,(User:irim),getListedStocksByMarkets,completed,in,7041,ms</code>)</p>
  
  <p>Loading that data with statement :</p>
  
  <p><code>records = LOAD 'myapp.log' using PigStorage(' ');</code></p>
  
  <p>did not achieve that because my tokens can be separated by several white space leading to several empty tokens. 
  PigStorage does not seem to support regexp delimiter (or at least I haven't succeeded configuring it that way).</p>
  
  <p>So my question : what would be the best way to get those tokens ?</p>
  
  <p>If I could remove empty elements from a relation I would be happy, is possible to do that with Pig ?</p>
  
  <p>For example starting from :</p>
  
  <p><code>(2012-08-01,,,INFO,,,(User:irim),,getListedStocksByMarkets,completed,in,7041,ms</code>)</p>
  
  <p>To get </p>
  
  <p><code>(2012-08-01,INFO,(User:irim),getListedStocksByMarkets,completed,in,7041,ms</code>)</p>
  
  <p>I'm trying another approach with <code>TextLoader</code> then <code>TOKENIZE</code> but I'm not sure it's the best strategy. 
  Maybe a User Load Function is a more natural choice ... </p>
  
  <p>Regards,</p>
  
  <p>Joel</p>