How did they assign absolute configuration to these cis and trans 2-methylcyclohexanols?
You have somehow decided that the front side of the ring is actually the rear side … or something. So your drawings show exactly the wrong stereochemistry. For the hydroxy-carbon, you thus would, in theory, get the incorrect assignment (S) but you use the rule you reference to reverse that into (R). Naturally, because your initial assertion of which atoms are in which plane was incorrect, your application of the rule was also incorrect. For the methyl carbon, your inverted ring reproducibly gives you the inverted assignment (R) and you do not apply any incorrect correction.


Figure 1: the molecule in question.

In figure 1, I have reproduced the molecule in the more ‘standard’ chair configuration which gives a better approximation of the bond angles. The wedged/bold portion of the ring is in front of the paper plane, the thin solid portion behind. I have also rotated the ring by $60°$ so that the hydroxy function is on the rightmost carbon atom for easier discussion.

It should be a rather simple exercise to determine that the hydroxy-carbon (carbon 1) indeed does have an (R) configuration: the hydroxy group has highest priority, the front side of the ring second-highest, the back side third and the hydrogen pointing downwards fourth. Therefore, we must look from above the ring plane to put the lowest substituent to the back; the rotation from hydroxy to methyl side to non-methyl side is thus clockwise, therefore (R).

The same analysis performed on carbon 2, the methylated one, should give you:


carbon 1 ($\ce{CHOH}$) is highest priority; going to the back right and slightly downwards
carbon 3 ($\ce{CH2}$) is second; going to the left, slightly downwards while staying in the plane
the methyl group is third; going up while staying in the plane
the unmarked hydrogen is fourth; this goes to the front right and slightly downwards.


Thus, we should look from inside the ring as the smallest substituent is pointing towards us. However, we can also look from the smallest substituent and invert the assignment; the way we see the three groups is again clockwise, so if we were to look from inside the ring (or behind it) we would see them anticlockwise, thus (S).

Note that the wedged and bold bonds — both in the paper’s drawing and in my figure — are not there to indicate a positioning from the stereocentre but to indicate which half of the ring is in front of the paper plane and which is behind. Thus, you cannot — like you did — assume the bolded bond to be in front; it actually is in your paper plane when you are looking at the two stereocentres on either side of it.