Am I able to wire a 3-Prong dryer outlet with 8/3 Romex?
You could. But don't!
Instead, replace the 3-wire outlet with a 4-wire outlet. It is much safer. There are no downsides. You will have to replace the dryer cord/plug with a 4-wire cord/plug, and remove the jumper connecting ground and neutral on the dryer (typically a green wire going from the neutral connection to some bare metal spot on the dryer).
You would have to do the same thing if it was a 10/3 as well. That is, assuming your 10/3 is (like the 8/3) 3 regular wires (typically black/red/white) + a ground wire.
If your 10/3 does not have a ground wire, ask for more help.