Is it appropriate (as a current student) to ask professors questions not directly related (but related enough) to course material?
As a former mathematician and academic, the interaction you describe strikes me as unusual. Certainly not the norm, and not in general what you should expect! Most of us, I suspect, learned most of the math we know via self-study, with the benevolent guidance of mentors as we went along. And as pedagogues we generally like helping people explore their interests in math, even when only very loosely connected with the matter at hand.

That being said, it is worth noting that university professors' time is pulled many ways, and in many cases their formal incentives are not at all aligned with spending their time opening the minds of random students on random math topics. At research universities, their career success depends on research productivity, and they may have little time for teaching to begin with, and helping students learn outside of the curriculum may not be high on their priorities. At more teaching-focused colleges, one hopes the desire to help would be strong(er), but teaching and administrative loads may be higher and they may feel their time is limited. In either case, if others are waiting for office hours they may feel those whose questions are directly related to the curriculum take precedence; and if their office hours are poorly attended (perhaps for a reason....), they may start feeling entitled to be using that time for lecture prep or something else unless someone comes with a directly relevant question. In my opinion, they shouldn't, but they may.

The basic advice of "find someone else who is less of a grumpster" applies. Beyond that, whether with this individual or someone else you approach, there are certainly things you can try to increase the positive response you'll get for "extracurricural" questions:


Make an appointment / "ask whether you can ask" and indicate roughly what you want to talk about, and recognize you know it's past the curriculum itself. Be willing to come another day.
Demonstrate enthusiasm and curiosity, avoid projecting entitlement. Professors want to stimulate the former, but we're pretty jaded about people who come across as if they feel they have a right to answers to whatever level of detail they feel like, or get irritated that the question they asked has a more complicated answer than they were clearly hoping for.
Do some reading/research beforehand, and indicate you have done so. There is a shift from highschool, where you are taught, to university (undergraduate and graduate), where you are being helped to learn. Show you are making that shift!


All of the above may well be things you are/were doing, and you just have a grumpy professor. Sorry about that. But I'm summarizing them anyway, since any academic or ex-academic will have stories of students whose keenness was infectious, as well as those whose non-stop, entitled questions were a pain in the neck!