def FULL_TEXT(val):
if isinstance(val, float):<EOL><INDENT>val = repr(val)<EOL><DEDENT>elif val in (None, '<STR_LIT>'):<EOL><INDENT>return None<EOL><DEDENT>elif not isinstance(val, six.string_types):<EOL><INDENT>if six.PY3 and isinstance(val, bytes):<EOL><INDENT>val = val.decode('<STR_LIT>')<EOL><DEDENT>else:<EOL><INDENT>val = str(val)<EOL><DEDENT><DEDENT>r = sorted(set([x for x in [s.lower().strip(string.punctuation) for s in val.split()] if x]))<EOL>if not isinstance(val, str):  <EOL><INDENT>return [s.encode('<STR_LIT:utf-8>') for s in r]<EOL><DEDENT>return r<EOL>
This is a basic full-text index keygen function. Words are lowercased, split
by whitespace, and stripped of punctuation from both ends before an inverted
index is created for term searching.