Uniform way of quantifying "branching" in nondeterministic, probabilistic, and quantum computation?
Nondeterministic computations can also be viewed as verification of claims using short proofs. That is, the class NTIME(t) can also be viewed as the class of languages $L$ such that a claim of the form $x \in L$ can be verified in time $t(|x|)$ by reading a short proofs. In this model, "quantifying the braching" is analogous to studying how short the proofs can be. While I don't know of papers that studied this question, this could give you a direction of where to look for them.
One paper that studied a related question in the context of interactive proofs is "On Interactive Proofs with Laconic prover", by Goldreich, Vadhan, and Wigderson:
http://www.wisdom.weizmann.ac.il/~oded/p_laconic.html

Regarding probabilistic computation: The amount of "branching" used in the computation is exactly the number of random bits/coin tosses that the algorithm uses. Quantifying this number and minimizing it is studied extensively in the area of "Derandomization". You can read about it in Chapters 20 and 21 of the Arora-Barak book (http://www.cs.princeton.edu/theory/index.php/Compbook/Draft) or in Chapter 8 of Goldreich's book (http://www.wisdom.weizmann.ac.il/~oded/cc-book.html).
It should also be mentioned that the common belief in this area is that $P=BPP$. If this is true, then the number of random bits a computation uses does not affect its power, as long as this number is polynomial.