'Return a list of s-expressions extracted from *text*.
For example:
>>> SExprTokenizer().tokenize(\'(a b (c d)) e f (g)\')
[\'(a b (c d))\', \'e\', \'f\', \'(g)\']
All parentheses are assumed to mark s-expressions.
(No special processing is done to exclude parentheses that occur
inside strings, or following backslash characters.)
If the given expression contains non-matching parentheses,
then the behavior of the tokenizer depends on the ``strict``
parameter to the constructor.  If ``strict`` is ``True``, then
raise a ``ValueError``.  If ``strict`` is ``False``, then any
unmatched close parentheses will be listed as their own
s-expression; and the last partial s-expression with unmatched open
parentheses will be listed as its own s-expression:
>>> SExprTokenizer(strict=False).tokenize(\'c) d) e (f (g\')
[\'c\', \')\', \'d\', \')\', \'e\', \'(f (g\']
:param text: the string to be tokenized
:type text: str or iter(str)
:rtype: iter(str)'

def tokenize(self, text):

    result = []
    pos = 0
    depth = 0
    for m in self._paren_regexp.finditer(text):
        paren = m.group()
        if (depth == 0):
            result += text[pos:m.start()].split()
            pos = m.start()
        if (paren == self._open_paren):
            depth += 1
        if (paren == self._close_paren):
            if (self._strict and (depth == 0)):
                raise ValueError(('Un-matched    close    paren    at    char    %d' % m.start()))
            depth = max(0, (depth - 1))
            if (depth == 0):
                result.append(text[pos:m.end()])
                pos = m.end()
    if (self._strict and (depth > 0)):
        raise ValueError(('Un-matched    open    paren    at    char    %d' % pos))
    if (pos < len(text)):
        result.append(text[pos:])
    return result
