Which side are these washers supposed to go?
It looks like the U-channel on that side came bent - If you compare it to the other side it seems to be bent quite a bit, judging from the pictures.  This is why the caster plate isn't hitting the other side, and why you though to put the washers behind the plate.  However, the washers are meant to go underneath the nut, not the plate.  I would suggest trying to bend the channel back into shape - try using a hammer and a block of wood - place the block of wood on the side of the channel, and then whack the block of wood with the hammer.  The block should help to minimize any distortion as you bend it back.