def get_text_tokenizer(query_string):
<EOL>split_pattern = re.compile('<STR_LIT>')<EOL>space_cleanup_pattern = re.compile('<STR_LIT>')<EOL>dash_cleanup_pattern = re.compile('<STR_LIT>')<EOL>keywords = [dash_cleanup_pattern.sub('<STR_LIT:->', space_cleanup_pattern.sub('<STR_LIT:U+0020>', t.strip('<STR_LIT>')))<EOL>for t in split_pattern.findall(query_string) if len(t.strip('<STR_LIT>')) > <NUM_LIT:0>]<EOL>include = [word for word in keywords if not word.startswith('<STR_LIT:->')]<EOL>exclude = [word.lstrip('<STR_LIT:->') for word in keywords if word.startswith('<STR_LIT:->')]<EOL>return include, exclude<EOL>
Tokenize the input string and return two lists, exclude list is for words that
start with a dash (ex: -word) and include list is for all other words