Are there any good out-of-the-box language models for python?
I also think that the first answer is incorrect for the reasons that @noob333 explained.
But also Bert cannot be used out of the box as a language model. Bert gives you the p(word|context(both left and right) ) and what you want is to compute p(word|previous tokens(only left contex)). The author explains here why you cannot use it as a lm.
However you can adapt Bert and use it as a language model, as explained here.
But you can use the open ai gpt or gpt-2 pre-tained models from the same repo
Here is how you can compute the perplexity using the gpt model.
import math
from pytorch_pretrained_bert import OpenAIGPTTokenizer, OpenAIGPTModel, OpenAIGPTLMHeadModel
# Load pre-trained model (weights)
model = OpenAIGPTLMHeadModel.from_pretrained('openai-gpt')
model.eval()
# Load pre-trained model tokenizer (vocabulary)
tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt')

def score(sentence):
    tokenize_input = tokenizer.tokenize(sentence)
    tensor_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])
    loss=model(tensor_input, lm_labels=tensor_input)
    return math.exp(loss)


a=['there is a book on the desk',
                'there is a plane on the desk',
                        'there is a book in the desk']
print([score(i) for i in a])
21.31652459381952, 61.45907380241148, 26.24923942649312