def sentence_to_token_ids(<EOL>sentence, vocabulary, tokenizer=None, normalize_digits=True, UNK_ID=<NUM_LIT:3>, _DIGIT_RE=re.compile(br"<STR_LIT>")<EOL>):
if tokenizer:<EOL><INDENT>words = tokenizer(sentence)<EOL><DEDENT>else:<EOL><INDENT>words = basic_tokenizer(sentence)<EOL><DEDENT>if not normalize_digits:<EOL><INDENT>return [vocabulary.get(w, UNK_ID) for w in words]<EOL><DEDENT>return [vocabulary.get(re.sub(_DIGIT_RE, b"<STR_LIT:0>", w), UNK_ID) for w in words]<EOL>
Convert a string to list of integers representing token-ids.

    For example, a sentence "I have a dog" may become tokenized into
    ["I", "have", "a", "dog"] and with vocabulary {"I": 1, "have": 2,
    "a": 4, "dog": 7"} this function will return [1, 2, 4, 7].

    Parameters
    -----------
    sentence : tensorflow.python.platform.gfile.GFile Object
        The sentence in bytes format to convert to token-ids, see ``basic_tokenizer()`` and ``data_to_token_ids()``.
    vocabulary : dictionary
        Mmapping tokens to integers.
    tokenizer : function
        A function to use to tokenize each sentence. If None, ``basic_tokenizer`` will be used.
    normalize_digits : boolean
        If true, all digits are replaced by 0.

    Returns
    --------
    list of int
        The token-ids for the sentence.