How exactly does 8PSK work on each I and Q leg?
You are mapping a group of 3 bits (an 8PSK symbol) to one of 8 coordinates in the I-Q plane.  I'm not an FPGA guy, but you can do this with a "switch" statement or a lookup table.  I don't know your particular mapping, but a straightforward mapping from bits to I &amp; Q values would be:

$$(0,0,0) \rightarrow (1, 0)$$
$$(0,0,1) \rightarrow (1/\sqrt{2},1/\sqrt{2})$$
$$(0,1,0) \rightarrow (0, 1)$$
$$(0,1,1) \rightarrow (-1/\sqrt{2},1/\sqrt{2})$$
$$(1,0,0) \rightarrow (-1, 0)$$
$$(1,0,1) \rightarrow (-1/\sqrt{2},-1/\sqrt{2})$$
$$(1,1,0) \rightarrow (0, -1)$$
$$(1,1,1) \rightarrow (1/\sqrt{2},-1/\sqrt{2})$$

Note that, in the above mapping, if you take entries 2, 4, 6, and 8, and ignore the least significant bit, you get a mapping for QPSK.