How to get a protein's dihedral angles ordered by variance?
If I understand you, I can't imagine how to solve the your problem with this approach. Also I don't get the physical meaning of the proposed solution.

If you have to deal with a molecule with more than 100 rotatable bonds, you are just in problems.
You could find the rotatable bonds with some tools (see for example OpenBabel). But the problem will just start.

At first, you can not just try one by one the $N$ angles ($\{\theta _i \}$) (by changing the angle value $n$ times and trying each of them), because for exploring all the conformational space you will just have $n^N$ points. And you just can't know a priori which are the most important angles (in case (I think not) that effectively exist the most important angles).

Also note that not every point ($\theta _1, \theta _2, \cdots,\theta _N)$ is acceptable, just because parts of the molecule can superpose with other parts.


  "Take a set of different conformations and use principal component analysis on the phase space consisting of the dihedral angles."


I think that this is not possible. The values of the rotatable angles will be just chosen by you.
Even more, suppose that you have two rotatable bonds very close each other, in a linear part of the molecule. If not a large superposition ocurrs, you will have no diference in the efect of them. But the effect of them will be very correlated. So, you can get useful info. with PCA.

A possible approach

As I know very few of biology, I would make a research about if there are prefered configuration, that are much more common than others (because of biological facts).

If you are forced to work with the entire configurational space, the computational approach that comes to my mind is:


Use very fast method for energy computations that allow you set the connectivity of the atoms. This will prevent changes of the covalent bonds due to superpositions.
Use the Monte Carlo method, this should explore the configurational space giving you a reasonable idea of this space.


Final note

I find extrange doing docking with a large protein and ¿without a ligand? If there is a ligand for binding to the protein that we are talking about, I won't take too much care of the protein. The scoring functions or energies are few accurate, I find hard to believe that just one found configuration would represent accurately the problem (considering the large degeneracies involved)