Shift-register seems to read wrong input data
The real reason might be just too slow signal at RCLK, or even on SCLK.
For example the TI 74HC595 has a maximum input transition rise or fall time of about 400ns at 6V or 500ns at 4.5V.
It means the signal may not spend more than the abovementioned time in the input threshold region between 0.5V and 1.5V, or there might be double clocking.
The RCLK will have very slow edge due to 10k resisor and 100nF capacitor. If I have it right, it takes about 270 microseconds. That is about 500 times too slow edge so it is definitely outside recommended operating conditions.
As the scope traces show, also the SCLK is driven with a relatively slow rising open-collector optocoupler output, it can be seen from the shark-fin like appearance. This clock should also be checked to be within required rise time.
The data should have ideally have fast edges too, as slow edges have also other implications, but at least it by itself does not cause double-clocking. The red data trace has also weird droop and goes negative which may indicate other problems in the circuit or measurements.