How to aggregate face embeddings of all photos of the same person?
This question is the first I've heard of FaceNet, but I don't think that the right solution to the question is to aggregate the face embeddings but to ask why you're using an SVM to classify the embeddings. Importantly, many SVM implementations of multiclass classification use a one-vs-rest method to train the classifiers -- if you're using a one-vs-rest implementation with 3000 labels, I suspect that this is the reason your training is taking so long.

You should look into how your implementation is training the classifier. Additionally, How large is your embedding size?