Substitute IDs to word tokens I writing a function in Python to substitute word tokens in a sentence with IDs.

The sentence is a list of tokens (list_of_tokens). 
The IDs are provided in a dictionary mapping tokens with an IDs (tokens2IDs_dict).

However, the functions I've written is not very efficient because of the for-loop. I am wondering if there is any more pythonic and efficient way to perform this mapping.

def tokens2IDs(list_of_tokens, tokens2IDs_dict):
    new_sentence = []
    for token in list_of_tokens:
        if len(token) == 0 or token == " ":
               continue
        new_sentence.append(tokens2IDs_dict.get(token, tokens2IDs_dict.get('UNK')))
    return new_sentence
There is nothing wrong in general with for loops. If you have to iterate over some elements there is no magic that can avoid this. So why is your function "not very efficient"? And why is this because of the for-loop?

However you asked for a pythonic solution. In Python explicit for loops are avoided by using list comprehension. Internally Python again loops over all elements, however slightly more efficient. Better readability is the other big gain.

So let's measure (I did some renaming for readability). I compare


your original explicit for loop
a 1:1 translation to a list comprehension
further improved by extraction of the default lookup
also apply Jean-François Fabres strip()


.

import timeit

def encode(tokens, dictionary):
    ids = []
    for token in tokens:
        if len(token) == 0 or token == " ":
            continue
        ids.append(dictionary.get(token, dictionary.get('UNK')))
    return ids

def encode1(tokens, dictionary):
    ids = [dictionary.get(t, dictionary.get("UNK")) for t in tokens if t and t != " "]
    return ids

def encode2(tokens, dictionary):
    unk = dictionary.get("UNK")
    ids = [dictionary.get(t, unk) for t in tokens if t and t != " "]
    return ids

def encode3(tokens, dictionary):
    unk = dictionary.get("UNK")
    ids = [dictionary.get(t, unk) for t in tokens if t.strip()]
    return ids

def test(tokens, dictionary, result, encoders):
    for encode in encoders:
        assert result == encode(tokens, dictionary)

def time(tokens, dictionary, encoders):
    print(tokens)
    print(encoders[0](tokens, dictionary))
    for encode in encoders:
        print(timeit.timeit(lambda: encode(tokens, dictionary)))

encoders = [encode, encode1, encode2, encode3]

tokens = ["asdf", "qwer", "yxcv", " ", ""]
dictionary = {"UNK": 0, "asdf": 1, "qwer": 2}
test(tokens, dictionary, [1, 2, 0], encoders)

string = "asdwerasfeqtdfgrtzdfhetzufgjhtzui"
tokens = list(string)
dictionary = {k: v for v, k in enumerate(["UNK"] + list(dict.fromkeys(tokens)))}
time(tokens, dictionary, encoders)

tokens = [" "] * len(string)
time(tokens, dictionary, encoders)

tokens = [""] * len(string)
time(tokens, dictionary, encoders)


results on my machine

['a', 's', 'd', 'w', 'e', 'r', 'a', 's', 'f', 'e', 'q', 't', 'd', 'f', 'g', 'r', 't', 'z', 'd', 'f', 'h', 'e', 't', 'z', 'u', 'f', 'g', 'j', 'h', 't', 'z', 'u', 'i']
[7, 9, 6, 11, 10, 12, 7, 9, 8, 10, 5, 14, 6, 8, 15, 12, 14, 1, 6, 8, 2, 10, 14, 1, 3, 8, 15, 4, 2, 14, 1, 3, 13]
7.497583819553256
5.152629452757537
3.7371353656053543
4.100516518577933
[' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ']
[]
1.744785476475954
1.0442649777978659
1.1422674069181085
1.721993056125939
['', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '']
[]
1.2569031789898872
0.6375910267233849
0.7521892953664064
1.699515514075756


which shows:


list comprehension is indeed significantly faster than the explicit for-loop
avoiding the evaluation of the default also has significant impact
strip() is a little slower than your explicit test (while more generic)
for the explicit tests the order of evaluation is important


Altogether this is doubling the speed, but there is no magic to find magnitudes.