How possible is it to do mathematical research outside academia?
Publishing
In mathematics, most people will judge your research on its merits, not based on your credentials or affiliations.  So this is not an issue, if you are able to generate significant results and write them up in a clear way.
Seminars
I have seen many members of the public regularly attend seminars.  I've never heard of them being kicked out.  It's not an issue.
Funding
The primary drawback is that you won't be paid for your research if you are just doing it in your spare time.  Unless you are already independently wealthy, that means you will have relatively little time for research since you need a day job.
It is of course possible to get grants to fund your research without being in academia; there are a number of private entities in the US funded entirely by grants, and many of them include mathematicians.  But you said spare time only.
Collaboration
Another drawback is that you won't be in close contact with other researchers by default.  It wouldn't be impossible to establish that contact, but again you must consider whether you will have the time.  And people won't take you seriously unless you have serious results.
Not getting a PhD
In mathematics, even more than in other fields, it is almost impossible to make a contribution without spending several years learning what is already known.  In principle you could do that without getting a PhD, but why would you?