How can I tokenize a text file with BERT or something similar?
With Huginface's Transformers, it should be doing with not much boilerplate.

A minimum example using PyTorch:

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple


Depending on how data you want to process, you might want to speed it up a little bit. In that case, you should do it batches which would require doing padding and passing the padding mask to the model.