Attending Conferences as an unaffiliated listener
Go for it ... probably.

There are no doubt aspects of conferences that are specific to individual fields, so my answer based on experience in Computer Science may not apply to Mathematics.  But I'm guessing the answer to this is somewhat general, in that some conferences are small (and may be largely limited to people who are actively participating/presenting) while others are larger.  If you pick a large enough conference, with many more attendees than presenters, no one will think twice of who you are or why you're there.