The holes in our ceiling junction box are not parallel to the walls. Do I have options other than removing the junction box?
You can try an adaptor (universal crossbar) (adjustable swivel crossbar)

Or just live with it as is?