How far can 3.3V UART go at 38400 baud?
To find the distance a signal can go the physical layer of communications needs to be specified.  RS-422 &amp; RS-485 define a physical layer that can be "looked up".  Descriptions such as UART do not define a physical layer and so are difficult to "lookup up" or comment on.

The cable used for long communications connections can play an important part in delivering a strong signal while mitigating the effects of electrical noise.  Registered Jacks sometimes use twisted pair cables to accomplish this.  But specifying an RJ12 Registered Jack does not ensure this type of cable is used. 

There are several other concerns when dealing with communications over cables.  Ground Loops being among them.  In extreme cases optical isolators are used to mitigate the effects of Ground Loops.

For this particular case, 0 to 3.3 V swing at 38400 baud over a 6 inch cable, it can only be said that it is a bit surprising that it does not work.  Consider the wiring is faulty or the signal is logically inverted. Then consider that the protocols at both ends do not match (such as parity, number of bits &amp; length of stop).  Finally consider the speed it too great for the hardware or software to handle.  


  On a broader scale, how do you predict usable cable length given a
  baud rate and voltage level?


It would be difficult. Many other factors affect the signal such as line, source and termination impedance.  Inspecting the signal with an oscilloscope at the source and destination may give some insight.