title: Filtering tokens from Scala Parser Combinators
body: <p>How do I filter the sequence of tokens coming from my Lexer to my Parser when using Scala parser combinators?</p>
  
  <p>Let me explain - suppose I have the fairly standard pattern of a Lexer (extending <code>StdLexical</code>) and a Parser (extending <code>StdTokenParsers</code>). The lexer turns a sequence of characters to a sequence of tokens, then the parser turns the sequence of tokens to an abstract syntax tree (of type <code>Expr</code>).</p>
  
  <p>I decide that some tokens, which could occur anywhere in the stream, I would like to have the option of filtering out, so I would like a function that would fit between the Lexer and Parser to remove these tokens. For example, I might want the lexer to tokenise comments, and then filter out these comments later. </p>
  
  <p>What is the best way of writing this filter? This could use the parser combinator idiom, but doesn't have to.</p>
  
  <p>Sample current code:</p>
  
  <pre><code> val reader = new PagedSeqReader(PagedSeq.fromReader(reader))
   val tokens = new MyParser.lexical.Scanner(reader)
   val parse = MyParser.phrase(parser)(tokens)
  </code></pre>
  
  <p>I would like to be able to write something like this: </p>
  
  <pre><code> val reader = new PagedSeqReader(PagedSeq.fromReader(reader))
   val tokens = new MyParser.lexical.Scanner(reader)
   val parse = MyParser.phrase(parser)(filter(tokens))
  </code></pre>