Taking a course on Pass / No Pass Basis for Grad School
I have been to Ph.D. admissions committees both as a grad student and as a faculty member. Generally speaking, if those courses are not very relevant to your major and your intended area of research, I wouldn't pay much attention to "Pass" grades (whereas low letter grades would probably stand out) We get plenty of domestic and foreign applicants with very diverse backgrounds and there are much more important things to look at in their application package.

I assume that the genesis of your position is that you want to be challenged in class, but not at the expense of risking a bad grade staining your otherwise impressive transcript in a subject that probably won't define your career. While this is certainly not the only approach, it seems sound thinking to me.

If they do affect your GPA in a negative way, however, I would consider a bit more carefully--perhaps not in a binary way, but in regards to how many courses to do this with.

By the way, remember that as a Ph.D. student you can also take courses outside your major you feel curious about. I certainly did that.