Spacer for non-drive side dropout
There are spacers that are sort of in the vein of what you describe. They look like this:



However, these are intended to be used as a matched set and aren't adjustable, so using just one of them probably wouldn't just do what you're wanting here. You could probaby file one down to make it do that, however.

Another thing to bear in mind here is that it's actually very common, especially in the bike boom era, for manufacturers to use the horizontal dropouts of their frames to give themselves room to be a little sloppy with the symmetry of the left and right chainstay/dropout assemblies. As long as the paths of the axle slots are parallel, having the start and end points be off by a few mm doesn't matter much. In other words there are a lot of horizontal dropout bikes where if, say, you pulled the axle all the way back in the dropouts on each side, the wheel would be off-center in the frame, and where using spacers like the pictured ones wouldn't work unless you nudged one ahead of the other the right amount.