How exactly does 8PSK work on each I and Q leg? I am working on making a modulator in VHDL and am implementing 8PSK. Currently, I have BPSK and QPSK looking good, and I have verified the spectrums. However, now that I am moving to 8PSK, I am stuck on how the data mux to the IQ legs should work?

Currently, my implementation is two-fold for QPSK. My first method splits the bits into the I and Q legs for even/odd bits (I am not pulse shaping, purely PCM), then I basically take this I or Q bit and xor it with the MSB of the signed DCO value, in-phase for even bits and quadrature for odd bits (I am using a digitally controlled oscillator as my carrier). The second method I have is I just generate the symbols for QPSK and feed my DCO with a phase shift at 45, 135, 225, and 315 (this saves logic as I only have to use 1 DCO). I know I can use the phase request for 8PSK as well, however, this method produces a lot of spurs. 

If I want to perform the following block diagram for 8PSK, how does this work? Am I accumulating 3 bits, or one symbol, for each leg, and performing a phase shift? I'm really struggling to figure out how to do this. I know I can just accumulate one symbol and send a phase shift to the DCO, but this doesn't use I and Q and feels like a crappy way of doing this, especially considering it doesn't work well when I tried it for QPSK. Here is the fundamental question, when I go from serial to parallel through the data MUX, what exactly is going onto the I and the Q leg? If it is still 1 bit, how is this any different than QPSK?
You are mapping a group of 3 bits (an 8PSK symbol) to one of 8 coordinates in the I-Q plane.  I'm not an FPGA guy, but you can do this with a "switch" statement or a lookup table.  I don't know your particular mapping, but a straightforward mapping from bits to I &amp; Q values would be:

$$(0,0,0) \rightarrow (1, 0)$$
$$(0,0,1) \rightarrow (1/\sqrt{2},1/\sqrt{2})$$
$$(0,1,0) \rightarrow (0, 1)$$
$$(0,1,1) \rightarrow (-1/\sqrt{2},1/\sqrt{2})$$
$$(1,0,0) \rightarrow (-1, 0)$$
$$(1,0,1) \rightarrow (-1/\sqrt{2},-1/\sqrt{2})$$
$$(1,1,0) \rightarrow (0, -1)$$
$$(1,1,1) \rightarrow (1/\sqrt{2},-1/\sqrt{2})$$

Note that, in the above mapping, if you take entries 2, 4, 6, and 8, and ignore the least significant bit, you get a mapping for QPSK.