Why don't contra-rotating propeller configurations with more than two propellers exist?
No, stacking more propellers will reduce efficiency. The case with two is an exception because the second propeller benefits from operating in the slipstream of the first. In total, both propellers accelerate the air backwards without adding swirl losses.

However, propeller efficiency goes down with increasing flow speed ahead of the propeller. This means that every other propeller operating in the accelerated flow of a preceding propeller will have lower efficiency. Generally, it will be better to spread the propellers out sideways than to put them in line.

Note that contra-rotating props are the exclusive realm of very powerful engines with very high disk loading. The most recent designs prefer to use more blades and avoid the heavy gearbox, but the efficiency of such props is poorer than that of propellers with lower disk loading and fewer blades. In a way, the efficiency of the eight-bladed propellers below is similar to that of four two-bladed propellers in sequence, but it is much easier to build with all blades and their pitch mechanism in one hub.


T-56 engines with Hamilton-Sundstrand NP2000 propellers on an LC-130. Photograph by Robyn Waserman, National Science Foundation (Date Taken: November 21, 2008)