How to fix rough opening of closet door
Either way you go here there's drywall and paint involved.

If it's not load bearing, just move the stud until you can fit the larger door. Barring miracles this will involve removed, replacement and refinishng of drywall.

If it's load bearing, add a stud on one side to make the new opening and drywall it in, since you can't cover it (or one on each side) with trim.