Is it possible to control and reverse the direction of a dc motor through a headphone socket?
Within reason, it is possible to drive almost anything from a headphone socket if you have some 'smarts' on the other side decoding the signal. As you have said, you'd prefer not to use the Arduino, but there are of course much smaller microcontroller systems if physical size is a problem.

For bidirectional control of a DC motor, you need at least a power supply and an H-Bridge. It would actually be possible (with perhaps some risk to your phone) to drive the 'A' channel on the H-bridge with your left audio output and the 'B' channel with your right audio output. This makes some assumptions which may or may not be true depending on your phone, for example, that your audio output is DC coupled (that is, it will pass a very low frequency signal). Of course, you would likely have to use a transistor on each input to shift the levels from ~1V P2P to ~5V P2P (if you are using a common H-bridge chip such as the L298). So you are looking at probably about half a dozen parts anyway including passives. 

Probably the best thing is to invest in doing it the 'right way' by devising some simple digital protocol which may be transmitted by your software and using a simple microcontroller on the device. It would be fairly straightforward to do with even the simplest 6-pin microcontroller.