Yes, language model can be trained on the text portion of patents downloaded from a patent database website, in fact many researchers are already experimenting with this. 

However, the diagram and image portions of patents could not be processed by a language model without some intermediate step, as language models can only take text as input and training data.  A different model would be required for images and diagrams, such as an image classification model or an image-to-text model.
spam fails_task lang_mismatch pii not_appropriate hate_speech sexual_content quality toxicity humor helpfulness creativity violence