Create similarity matrix
You have two scenarios here.


The vectors with the same labels are close to each other.


Then, what you can do is perhaps use the distance between the two closest/furthest elements in the groups with label x and the group with label y. 

You could also create an average vector for each label, and then simply get the distance between these.


Your vectors are not close to each other


This is not an issue. Here, what you'll want to see, is if for instance, for each vector with label x, there is a vector with label y close to it.

To do that, you could use the Earth Mover's Distance which gives you a single score when you are trying to see how far two "groups of things" are.