Dihedral group D4 composition with custom labels
Ruby, 18 bytes



-&gt;a,b{a+b*~0**a&amp;7}


Ungolfed

-&gt;a,b{ (a+b*(-1)**a) % 8}  
# for operator precedence reasons, 
#-1 is represented as ~0 in the golfed version 


Try it online!

Uses the following coding numbers 0 to 7

In order native to the code:

Native     Effect                    Codes per
Code                                 Question
0          rotate 0 anticlockwise    1C
1 /        flip in y=x               7E
2 /|       rotate 90 anticlockwise   2O
3 /|/      flip in x axis            5U
4 /|/|     rotate 180 anticlockwise  3M
5 /|/|/    flip in y=-x              8R
6 /|/|/|   rotate 270 anticlockwise  4P
7 /|/|/|/  flip in y axis            6T


In order per the question

Native     Effect                    Codes per
Code                                 Question
0          rotate 0 anticlockwise    1C
2 /|       rotate 90 anticlockwise   2O
4 /|/|     rotate 180 anticlockwise  3M
6 /|/|/|   rotate 270 anticlockwise  4P
3 /|/      flip in x axis            5U
7 /|/|/|/  flip in y axis            6T
1 /        flip in y=x               7E
5 /|/|/    flip in y=-x              8R


Explanation

/ represents a flip in the line y=x and | represents a flip in the y axis.

It is possible to generate any of the symmetries of the group D4 by alternately flipping in these two lines For example / followed by | gives /| which is a rotation of 90 degrees anticlockwise.

The total number of consecutive flips gives a very convenient representation for arithmetic manipulation.

If the first move is a rotation, we can simply add the number of flips:

Rotate 90 degrees   +  Rotate 180 degrees = Rotate 270 degrees
/|                     /|/|                 /|/|/|

Rotate 90 degress   +  Flip in y=x        = Flip in x axis   
/|                    /                     /|/


If the first move is a reflection, we find we have some identical reflections / and | symbols next to each other. As reflection is self inverse we can cancel out these flips one by one. So we need to subtract one move from the other

Flip in x axis     +  Flip in y=x        = Rotate 90 degrees
/|/                   /                    /|/ / (cancels to) /|

Flip in x axis     +  Rotate 90 degrees  = Flip in y=x
/|/                   /|                   /|/ /| (cancels to ) /