def ngrams(tokens, min_n, max_n):
all_ngrams = list()<EOL>n_tokens = len(tokens)<EOL>for i in xrange(n_tokens):<EOL><INDENT>for j in xrange(i + min_n, min(n_tokens, i + max_n) + <NUM_LIT:1>):<EOL><INDENT>all_ngrams.append("<STR_LIT:U+0020>".join(tokens[i:j]))<EOL><DEDENT><DEDENT>return all_ngrams<EOL>
Generates ngrams(word sequences of fixed length) from an input token sequence.
tokens is a list of words.
min_n is the minimum length of an ngram to return.
max_n is the maximum length of an ngram to return.
returns a list of ngrams (words separated by a space)