Can you move backwards 4 spaces right outside of the safety zone?
Yes. From the rules, under "Moving Backwards":


  If you have successfully moved a pawn backwards at least two spaces beyond your own START space, you may, on a subsequent turn, move into your own SAFETY ZONE without moving all the way around the board.