def basic_tokenize(input_str, lang):
    '\n    The function is used to do some basic tokenization\n\n    Args:\n        input_str: The input string\n        lang: Language of the input string\n    Return: a list of tokens of the input string\n    '
    if (lang == constants.ENGLISH):
        return word_tokenize(input_str)
    return input_str.strip().split(' ')
The function is used to do some basic tokenization

Args:
    input_str: The input string
    lang: Language of the input string
Return: a list of tokens of the input string
def basic_tokenize(input_str, lang):
    '\n    The function is used to do some basic tokenization\n\n    Args:\n        input_str: The input string\n        lang: Language of the input string\n    Return: a list of tokens of the input string\n    '
    if (lang == constants.ENGLISH):
        return word_tokenize(input_str)
    return input_str.strip().split(' ')