Splitting each word using longest substring that matches in a vocab/dictionary
The following is a basic syntax and usage pass without looking into your algorithm in too much depth. I'll first show the suggested code and then highlight significant differences where it can offer improvements on the original.

#!/usr/bin/env python3

from typing import Iterable, Set


def tokenize(
    text: str,
    vocab: Set[str],
    max_input_chars_per_word=10,
    unk_token="[UNK]"
) -&gt; Iterable[str]:
    for token in text.split():
        n = len(token)
        if n &gt; max_input_chars_per_word:
            yield unk_token
            continue

        start = 0
        sub_tokens = []

        while start &lt; n:
            end = n
            cur_substr = None
            while start &lt; end:
                substr = token[start:end]
                if start &gt; 0:
                    substr = "##" + substr
                if substr in vocab:
                    cur_substr = substr
                    break
                end -= 1
            if cur_substr is None:
                yield unk_token
                break
            sub_tokens.append(cur_substr)
            start = end
        else:
            yield from sub_tokens


def test():
    vocab = {"the", "go", "##es", "to", "eat", "pum", "##pkins", "of", "##gos", "##stein"
             "#400", "1", "boredom", "##folk", "man",
             "##go", "out", "folks", "##0",
             "un", "##aff", "##able"}

    s = "the unaffable folks goes to eat 1400 folkspkinsgosgo pumpkins and 10 mangos out of boredom"

    result = tokenize(s, vocab)
    assert list(result) == [
        'the',
        'un',
        '##aff',
        '##able',
        'folks',
        'go',
        '##es',
        'to',
        'eat',
        '[UNK]',
        '[UNK]',
        'pum',
        '##pkins',
        '[UNK]',
        '1',
        '##0',
        'man',
        '##gos',
        'out',
        'of',
        'boredom']


test()


Rationale


Add PEP484 type hints such as Iterable for the return of your function.
Require that you accept a set for vocab. All you do on it is membership testing, i.e. if substr in vocab:, so a set will be much faster.
Make this function a generator and yield instead of building up an output_tokens list.
Store the token length in a variable such as n, considering you need it so often.
Do not split out token into chars, and do not call join. All you need is the token string.
Do not track an is_bad flag. Use a for/else to detect whether a break occurred.
Use your test data in actual assert tests.