How do RIM and SIM instructions of microprocessor 8085 work?
The 8085 RIM (Read Interrupt Mask) and SIM (Set Interrupt Mask) instructions are confusing because they have two completely unrelated functions: serial data and interrupt masking.

Here is the format of the SIM instruction.  The SIM instruction uses the data in the accumulator as follows:



D7-D6 - The left two bits are related to the serial interface.  When D6 (SDE-Serial Data Enable) is 1, then whatever is in D7 (SOD-Serial Data Output) is written to the serial data output (pin 4 of the 8085).  If D6=0, nothing is written.  This allows a SIM instruction to be executed altering interrupt masks without affecting serial data.

Bit D5 is not used.

Bit D4 (R 7.5-Reset RST 7.5) This bit allows the SIM instruction to reset the interrupt pending flag indicated by bit D6 in the RIM instruction layout. The 7.5 interrupt can indicate that it is pending via the RIM instruction even though it is masked off. This bit allows that pending request to be reset.

Bit D3 (MSE-Mask Set Enable) is like SDE -- it indicates whether the lower three bits (D2-D0) are ignored or not.  This allows the serial data output to occur without affecting the interrupt masks.  If a SIM is executed with this bit low, the condition of the mask bits will not change. If a SIM is executed with this bit set high, the mask bits will be set according to the lower three bits of the accumulator. 

Bits D2-D0 (RST 7.5 Mask, RST .5 Mask, RST 5.5 Mask) These are the interrupt masks for the 8085 interrupts 7.5, 6.5, and 5.5.  If the corresponding bit is 0, the interrupt is enabled.  If the bit is 1, the interrupt is masked (ignored).

Here is the format of the RIM instruction.  The RIM instruction reads the following bits into the accumulator:



Bit D7 (SID-Serial Input Data) This is the input pin of the serial data interface which is connected to pin 5 of the 8085, and indicates the high/low status of that pin.

Bits D6-D4 (I 7.5, I 6.5, I 5.5) These bits indicates that an interrupt is pending for these three 8085 interrupts 7.5, 6.5, and 5.5.   If interrupts 5.5 or 6.5 have been masked off by bits D0 or D1, bits D4 and D5 will not be set. Bit D6, which corresponds to the 7.5 interrupt, will be set on to indicate that an interrupt 7.5 was requested, even if it was masked off.

Bit D3 (IE-Interrupt Enable) This bit indicates whether interrupts are enabled (1) using the EI (Enable Interrupts) instruction, or disabled (0) using the DI (Disable Interrupts) instruction.

Bits 2-D0 (M 7.5, M6.5, M5.5) Mask status of interrupts 7.5, 6.5, and 5.5.  Corresponds to bits D2-D0 of the SIM instruction.  1 if masked, 0 if enabled.

So the SIM and RIM instructions are typically used to either output to or input from 8085 serial interface, or enable/disable/read the interrupt masks for interrupts 7.5, 6.5, 5.5, but usually not at the same time.

Note: the 8085 serial interface is somewhat of a misnomer; it is just two pins, input and output, that have related bits in the RIM and SIM instructions (rather than a port).  All of the timing has to be done by the programmer -- they are not UARTs.  Also, the input (pin 5) doesn't generate an interrupt unless tied into one of the other interrupt lines.