What could be causing the washing machine waste pipe to overflow?
The first thing to do, is to figure out how this drain is vented. If you've got a big slug of water from the washer moving through the pipe, you've got to equalize the pressure in front of and behind the water. So you'll want to make sure the drain is properly vented. 

Next, I'd probably remove all that extra plumbing.  With three 90° bends like that, you're making it more difficult for the water to move through the pipes.

Cut out the extra plumbing, and connect the trap assembly directly to the trap arm. Cut the plumbing at the orange line in the image below, then connect the trap and standpipe to the newly cut pipe. This eliminates all the extra bends, and should allow the water to drain better.



I'd also install a shallower P-trap. The shallower trap will hold less water, and allow better flow. As it's currently plumbed, you've got probably about 4-5" of water sitting in the trap. 



Lastly, you may be able to make the standpipe longer. Check your local codes for the maximum length, as some say maximum 30", while others allow up to 42" pipes. (the standpipe is the vertical pipe coming up out of the trap). A longer standpipe obviously holds more water before overflowing, so it can provide a bit of extra time for the water to drain.