The embeddings for words and POS tags were pre-trained on a large unannotated corpus consisting of the first 1 billion characters from Wikipedia. 
 Question: Do they use pretrained models as part of their parser?
Yes