How common are shortlist notifications for math postdocs?
It's not common to officially notify applicants that they are on a short list.  If you find out, it will probably be via backchannel communications through your advisor.  That's the best way to gather information, since your advisor's contacts may be more open with him/her than they would be with you, and your advisor can put in a good word for you at the same time.