How to find the most unique vectors in a set?
I suspect your problem might be NP-hard (by reduction from Clique), so it might be hard to find an efficient algorithm for it.

One heuristic is to use the farthest-point first method, where at each stage you pick the point whose angle to the others already picked is as large as possible.  In other words, suppose we have already picked a set $S$ of vectors.  Now we find the vector $x$ that maximizes $\min \{\text{angle}(x,s) : s \in S\}$, add it to $S$, and repeat.  This most likely won't give the optimal set of vectors, but it might give something that is somewhat close to optimal.

Another possible approach is to use binary search and some kind of clique-finding algorithm.  Let $t$ be a threshold.  Form an undirected graph $G$ where each vector is a vertex in the graph; and you have an edge $(u,v)$ if the angle between the two vectors $u,v$ is at least $t$.  Then if you can find a clique of size $k$ (where $k$ is the desired size of your subset), then you've found a subset of vectors that are all at angle $\ge t$ from each other.  Now repeat this, using binary search on $t$ to find the largest $t$ for which you can find a $k$-clique.  In this way you might be able to take advantage of methods in the literature for finding $k$-cliques.  (For instance, one simple preprocessing step you can do to reduce the size of the graph is:  delete any vertex whose degree is less than $k$, and repeat until convergence.)