How to align PVC pipe to metal electrical box
You can make a transition with a piece of carflex (LFNC) or sealtite (LFMC) to make a flexible offset in your conduit. Use a PVC male adapter and a sealtite connector to make the transition. Some hardware stores will already have small piece of flex premanufactured. The only problems with this is sometimes this can look pretty ugly, and pulling through may get a little tougher. Be sure and bring along some pulling lubricant to help.

Good luck.