Is there a common mathematical symbology for collections?
Specification languages such as Z have a vast range of data types and operations on them and, at least within that community, these are standardised. This tutorial gives a great overview. Other specification languages exist, such as B and Alloy which may also satisfy your requirements.