Can I extend a single twisted pair DSL line using a coaxial cable?
Can I connect the phone line to the coaxial cable, and plug in my
  modem at the other end?


The conservative answer has to be "no" - phone lines are "balanced" in that the impedance presented by each wire to ground is constant. Coax cable is "unbalanced" and the impedance presented by the screen can be vastly different to the impedance presented to ground by the inner core.

The telephone cable's natural characteristic impedance is around 600 ohm whereas the coax cable might be 50 ohms or 75 ohms and it could cause data symbol reflections and therefore an excessive bit error rate when attached to a system expecting 600 ohm.

However, being less conservative, if the coax cable is short (say no more than a few metres) then you might get away with it being inserted.