def data_to_token_ids(<EOL>data_path, target_path, vocabulary_path, tokenizer=None, normalize_digits=True, UNK_ID=<NUM_LIT:3>,<EOL>_DIGIT_RE=re.compile(br"<STR_LIT>")<EOL>):
if not gfile.Exists(target_path):<EOL><INDENT>tl.logging.info("<STR_LIT>" % data_path)<EOL>vocab, _ = initialize_vocabulary(vocabulary_path)<EOL>with gfile.GFile(data_path, mode="<STR_LIT:rb>") as data_file:<EOL><INDENT>with gfile.GFile(target_path, mode="<STR_LIT:w>") as tokens_file:<EOL><INDENT>counter = <NUM_LIT:0><EOL>for line in data_file:<EOL><INDENT>counter += <NUM_LIT:1><EOL>if counter % <NUM_LIT> == <NUM_LIT:0>:<EOL><INDENT>tl.logging.info("<STR_LIT>" % counter)<EOL><DEDENT>token_ids = sentence_to_token_ids(<EOL>line, vocab, tokenizer, normalize_digits, UNK_ID=UNK_ID, _DIGIT_RE=_DIGIT_RE<EOL>)<EOL>tokens_file.write("<STR_LIT:U+0020>".join([str(tok) for tok in token_ids]) + "<STR_LIT:\n>")<EOL><DEDENT><DEDENT><DEDENT><DEDENT>else:<EOL><INDENT>tl.logging.info("<STR_LIT>" % target_path)<EOL><DEDENT>
Tokenize data file and turn into token-ids using given vocabulary file.

    This function loads data line-by-line from data_path, calls the above
    sentence_to_token_ids, and saves the result to target_path. See comment
    for sentence_to_token_ids on the details of token-ids format.

    Parameters
    -----------
    data_path : str
        Path to the data file in one-sentence-per-line format.
    target_path : str
        Path where the file with token-ids will be created.
    vocabulary_path : str
        Path to the vocabulary file.
    tokenizer : function
        A function to use to tokenize each sentence. If None, ``basic_tokenizer`` will be used.
    normalize_digits : boolean
        If true, all digits are replaced by 0.

    References
    ----------
    - Code from ``/tensorflow/models/rnn/translation/data_utils.py``