Having a retired professor write my letter of recommendation,
So, I wonder whether anyone external to our own math department would even know who he is - or whether admissions committees even value his opinions about grad students, at this point in his career.


First, it's not necessary or even expected that admissions committees will know (of) all or even any of your letter writers.  Second, you will be getting several letters, so they do not all need to come from "research active" professors.  Third, there's no reason to undervalue someone's opinion because they've retired.

Yes, it's a plus if you can get letters from people who are well known, but the most important thing is to get letters from people who know you best, so I don't see a serious concern from what you've told us.  (Though now that he's retired, it's possible he is harder to contact or less inclined to write letters.)