vector_space_model/Int/VSM-master/vsm.py
def tokenize(document):
    """Returns a list whose elements are the separate terms in
    document.  Something of a hack, but for the simple documents we're
    using, it's okay.  Note that we case-fold when we tokenize, i.e.,
    we lowercase everything."""
    terms = document.lower().split()
    return [term.strip(characters) for term in terms]
Returns a list whose elements are the separate terms in
    document.  Something of a hack, but for the simple documents we're
    using, it's okay.  Note that we case-fold when we tokenize, i.e.,
    we lowercase everything.
Returns a list whose elements are the separate terms in
document.  Something of a hack, but for the simple documents we're
using, it's okay.  Note that we case-fold when we tokenize, i.e.,
we lowercase everything.
Returns a list whose elements are the separate terms in document . Something of a hack but for the simple documents we ' re using it ' s okay . Note that we case - fold when we tokenize i . e . we lowercase everything .
def tokenize(document):
    terms = document.lower().split()
    return [term.strip(characters) for term in terms]
def tokenize ( document ) : terms = document . lower ( ) . split ( ) return [ term . strip ( characters ) for term in terms ]
Returns a list whose elements are the separate terms in
document.
Returns a list whose elements are the separate terms in document .
"""Returns a list whose elements are the separate terms in
    document.  Something of a hack, but for the simple documents we're
    using, it's okay.  Note that we case-fold when we tokenize, i.e.,
    we lowercase everything."""
document