Did I remove more caulk then I should have at shower door frame end?
I think squeezing in enough caulk will fill the gap.  You can tape part of the opening to seal it so that you can build a bit of pressure up in there to fill all the crevasses.  It’s important to seal it from the inside too if you can, however keep an eye on the area for a couple weeks, especially right after you do the repair.