What algorithm should I use to classify documents?
Text approach:

Use LDA (Latent Dirichlet Allocation). LDA is unsupervised. Feed it in corpuses of text from the various documents (i.e. OCR them and feed LDA the results of OCR). It will then cluster them based on the contents of the text (with or without stop words - at your discretion). If possible, you could do a supervised approach of using a bag-of-words and any classifier such as an SVM or Random Forest.

Image Approach:

Use a CNN (convolutional neural network) and train it on images of the various vendors. If you don't have this class discrimination, and can't get it, then use an unsupervised approach such as an autoencoder and then cluster the points in the lower-dimensional autoencoder feature space.