How to diagnose and fix apartment AC that makes intermittent whooshing noise
"If I remove remove the filter on the offending side, it doesn't
  happen."


This tells us that it is an Airflow Issue and the culprit is in the filter mechanics. 

I am guessing you recently changed filters - perhaps to a new brand or type. Is the filter installed in the correct direction of Airflow - an Arrow on the side should point the way..

I am thinking your filters are the wrong type. The filter is not allowing enough flow (apparently a tight fit) and as the suction (vacuum) builds up the filter moves and eventually it is in a position to allow more air flow and you get the whooshing sound of air passing either around the filter or surprisingly through it.

You can verify the filter brand and type recommendations of your unit - or compare to non offending side - perhaps non-offending side fits loosely and offending side fits more tightly in the filter slots..??