Air Conditioner Draws a Lot of Outside Air When Door Opens
The local A/C contractor performed a door blower test prior to our upgrade to a heat pump. The particular door opens to the attached/integrated garage and the outside lift door was open.

A manometer in the fan assembly showed a certain pressure differential. Had it been particularly excessive, it would have indicated that air was being pulled from the attic through the ducting. I can't recall the figures, but we had all new ducting installed and the test was improved on the post-install check-out.

In your case, I'd suspect your concerns are valid, but it might not be leak related.

The primary return, often with filter, pulls air from all the other returns, which in my house, dumps into the main living area. Prior to the upgrade, the bedroom and bathroom doors did not have grill or grate type returns. If any of those doors were closed, those rooms could have "pressurized" and prevented return air from reaching the main return.

Opening an outside door might have then created the inrush you've described.

Alternatively, the pressurized air from the handler could be leaking, but would the leaks be of such volume that it would match the volume of air during the inrush?

I suppose an easy enough test would be to tape covers over your venting to see if the door air rush still happens. If all the vents are sealed or otherwise closed and there are no leaks, the air handler has no place to push the air and would have no need to pull air from the opening door.