def tokenise_directive(self, dirtext):
    'Called by the private _parse() method when it encounters a\n        DIRECTIVE token in the list provided by the split_text() or\n        interpolate_text() methods.\n\n        The method splits the directive into individual tokens as\n        recognised by the parser grammar (see template.grammar for\n        details).  It constructs a list of tokens each represented by 2\n        elements, as per split_text() et al.  The first element contains\n        the token type, the second the token itself.\n\n        The method tokenises the string using a complex (but fast) regex.\n        For a deeper understanding of the regex magic at work here, see\n        Jeffrey Friedl\'s excellent book "Mastering Regular Expressions",\n        from O\'Reilly, ISBN 1-56592-257-3\n\n        Returns the list of chunks (each one being 2 elements) identified\n        in the directive text.\n        '
    tokens = []
    for match in GRAMMAR.finditer(dirtext):
        for (indices, method) in self.tokenize:
            if match.group(indices[0]):
                tokens.extend(method(*list(map(match.group, indices))))
                break
    return tokens
Called by the private _parse() method when it encounters a
DIRECTIVE token in the list provided by the split_text() or
interpolate_text() methods.

The method splits the directive into individual tokens as
recognised by the parser grammar (see template.grammar for
details).  It constructs a list of tokens each represented by 2
elements, as per split_text() et al.  The first element contains
the token type, the second the token itself.

The method tokenises the string using a complex (but fast) regex.
For a deeper understanding of the regex magic at work here, see
Jeffrey Friedl's excellent book "Mastering Regular Expressions",
from O'Reilly, ISBN 1-56592-257-3

Returns the list of chunks (each one being 2 elements) identified
in the directive text.
def tokenise_directive(self, dirtext):
    'Called by the private _parse() method when it encounters a\n        DIRECTIVE token in the list provided by the split_text() or\n        interpolate_text() methods.\n\n        The method splits the directive into individual tokens as\n        recognised by the parser grammar (see template.grammar for\n        details).  It constructs a list of tokens each represented by 2\n        elements, as per split_text() et al.  The first element contains\n        the token type, the second the token itself.\n\n        The method tokenises the string using a complex (but fast) regex.\n        For a deeper understanding of the regex magic at work here, see\n        Jeffrey Friedl\'s excellent book "Mastering Regular Expressions",\n        from O\'Reilly, ISBN 1-56592-257-3\n\n        Returns the list of chunks (each one being 2 elements) identified\n        in the directive text.\n        '
    tokens = []
    for match in GRAMMAR.finditer(dirtext):
        for (indices, method) in self.tokenize:
            if match.group(indices[0]):
                tokens.extend(method(*list(map(match.group, indices))))
                break
    return tokens