What size door jamb should I order for a non-standard wall thickness?
I'm not sure that I fully understand your situation, but from what I think you're describing, you should be able to inset one side of the door frame without the flange by 11/16 and add a spacer or "extension" to bring the frame's edge flush with the wall's surface--and then attach molding, etc. on top of that as usual.

I've done that with a garage door that I installed one time and lived in houses with non-standard thickness walls and that's how it was done on those too.