Python regex -- extraneous matchings
Question: I want to split a string using `-`, `+=`, `==`, `=`, `+`, and white-space as
delimiters. I want to keep the delimiter unless it is white-space.

I've tried to achieve this with the following code:

    
    
    def tokenize(s):
      import re
      pattern = re.compile("(\-|\+\=|\=\=|\=|\+)|\s+")
      return pattern.split(s)
    
    print(tokenize("hello-+==== =+  there"))
    

I expected the output to be

    
    
    ['hello', '-', '+=', '==', '=', '=', '+', 'there']
    

However I got

    
    
    ['hello', '-', '', '+=', '', '==', '', '=', '', None, '', '=', '', '+', '', None, 'there']
    

Which is almost what I wanted, except that there are quite a few extraneous
`None`s and empty strings.

Why is it behaving this way, and how might I change it to get what I want?


Answer: [re.split](http://docs.python.org/2/library/re.html#re.split) by default
returns an array of the bits of strings that are in between the matches: (As
@Laurence Gonsalves notes, this is its main use.)

    
    
    ['hello', '', '', '', '', '', '', '', 'there']
    

Note the empty strings in between `-` and `+=`, `+=` and `==`, etc.

As the docs explain, because you're using a capture group (i.e., because
you're using `(\-|\+\=|\=\=|\=|\+)` instead of `(?:\-|\+\=|\=\=|\=|\+)`, the
bits that the capture group matches are interspersed:

    
    
    ['hello', '-', '', '+=', '', '==', '', '=', '', None, '', '=', '', '+', '', None, 'there']
    

`None` corresponds to where the `\s+` half of your pattern was matched; in
those cases, the capture group captured nothing.

From looking at the docs for re.split, I don't see an easy way to have it
discard empty strings in between matches, although a simple list comprehension
(or [filter](http://docs.python.org/2/library/functions.html#filter), if you
prefer) can easily discard `None`s and empty strings:

    
    
    def tokenize(s):
      import re
      pattern = re.compile("(\-|\+\=|\=\=|\=|\+)|\s+")
      return [ x for x in pattern.split(s) if x ]
    

**One last note** : For what you've described so far, this will work fine, but
depending on the direction your project goes, you may want to switch to a
proper parsing library. [The Python
wiki](http://wiki.python.org/moin/LanguageParsing) has a good overview of some
of the options here.

