Is it frowned upon to go to office hours as a graduate student?
Grad students who come to office hours with good questions that show serious engagement with the material (e.g. attempts to solve the problem themself) are generally seen as mature, hard-working students.

It is also a good way to get to know faculty in your department, which can be useful e.g. if you are looking for a research assistantship, or recommendation letters, etc.

Personally, when I teach, I like when students come to office hours with good questions - it helps me see how students are understanding the lesson, where common points of friction are, and thereby improves the quality of my instruction. It also helps them do better in the course, which is of course a goal we have in common. But, I enjoy teaching and engaging with students. Some faculty in my department don't like teaching, and don't like office hours - but even those faculty members don't look down on students who make (good) use of them.