Downsides of switching Master to Simple recovery?
It does not really make sense to have the master database in FULL recovory. The installation process leaves the database in SIMPLE recovery, so this means that somebody changed it afterwards because they could. 

I do not think that you perform operations on the master database that require point-in-time recovery and if you do I strongly suggest that you rethink this strategy. Also, take into account that a database in FULL recovery needs to back up its log regularly in order to avoid filling the log files. Is this happening? How are you planning to restore these log backups in case?

That said, I would change it back to SIMPLE without worrying too much about performance hits.