Classifying objects based of a varying number of the same type of feature vector for each object For a congressional session, I have created a doc2vec model of speeches made. Using the vectors from this model, I have a dataset of each congressperson, their political affiliation, and a list of the vector representations of each speech they made. Each of these document vector representations is a 300 element vector.

I am now trying to classify each congressperson by party using these document vectors representing their speeches. So far I have tried using the mean vector for each speaker, but I was looking for ways to use the whole set of vectors.
If I did not get you right please comment me. I would not go for doc2vec as you do not want to discriminate docs but persons. So better to concatenate speeches of each person to a single document and then feed it to word2vec/doc2vec (I assume you would like to use ANNs otherwise there are other options e.g. TF-IDF, etc.) In this manner each person will have a 300-d feature vector including a everything. (I would still try TF_IDF and CountVectorizer as well!)

If you insist on your current option, you may discard person info (you want to get a speech and say what kind of political-party the speaker is coming from, right?!) and set up your data as a 300-d feature vector for each speech whose tag is a political view (instead of taking mean of all speeches of 1 person). Then your feature vectors are all the same size (in this approach, each person might have several entry in the data which may bias the data so you may add a column including the speaker e.g. "speaker a", "speaker b", etc. Here you end up with a 301-d data).

Hope it helped. Good luck!