def detokenize(tokens):
text = '<STR_LIT:U+0020>'.join(tokens)<EOL>step0 = text.replace('<STR_LIT>', '<STR_LIT>')<EOL>step1 = step0.replace("<STR_LIT>", '<STR_LIT:">').replace("<STR_LIT>", '<STR_LIT:">')<EOL>step2 = step1.replace("<STR_LIT>", "<STR_LIT>").replace("<STR_LIT>", "<STR_LIT>")<EOL>step3 = re.sub(r'<STR_LIT>', r"<STR_LIT>", step2)<EOL>step4 = re.sub(r'<STR_LIT>', r"<STR_LIT>", step3)<EOL>step5 = step4.replace("<STR_LIT>", "<STR_LIT:'>").replace("<STR_LIT>", "<STR_LIT>").replace("<STR_LIT>", "<STR_LIT>").replace("<STR_LIT>", "<STR_LIT>")<EOL>step6 = step5.replace("<STR_LIT>", "<STR_LIT>")<EOL>return step6.strip()<EOL>
Detokenizing a text undoes the tokenizing operation, restores
punctuation and spaces to the places that people expect them to be.
Ideally, `detokenize(tokenize(text))` should be identical to `text`,
except for line breaks.