How are office hours for PhD classes different from office hours for undergraduate classes?
At every level of education, the purpose of office hours is to give students an open-ended opportunity to discuss the material that they are learning with their instructors, and particularly to get help and clarification in their understanding.

As an instructor, I found it greatly frustrating when students (or any level) reduce this opportunity into simply asking for help on homework questions.  I believe there are many other instructors who feel the same. In fact, I felt so strongly about this that I personally had a policy of refusing to help with homework questions per se, and would instead offer to work a related but different problem in order to try to get the student to learn the material rather than just solve the problem.

I find this attitude most widespread amongst undergraduates, but many graduate students have it to.  Hopefully, you are actually interested in the material that you are learning, rather than just your grade.  That attitude, then, can easily translate into requests for help that are about the material, rather than the problems.  For example, consider:


"Can you help me with problem 5?" vs.
"Problem 5 made me realize I don't really understand Urysohn's Lemma.  Can you help me understand that better?"


Whether graduate or undergraduate, if you ask questions of the second sort, any good instructor who cares about their students' learning is likely to appreciate it.