What rules would allow a pilot to refuse a passenger's boarding?
The regulation that springs immediately to mind is FAR 91.17(b):


  Except in an emergency, no pilot of a civil aircraft may allow a person who appears to be intoxicated or who demonstrates by manner or physical indications that the individual is under the influence of drugs (except a medical patient under proper care) to be carried in that aircraft.


i.e. "If you get yourself blind stinking drunk at the airport bar, or you've been doing lines of cocaine in the airport bathroom you are not getting on the plane because the FAA says I can't let you."



For other cases you need to head over to 49 USC 44902, which deals with Refusal to transport passengers and property for "air carriers" (FAR part 121, and I believe 135 operations are covered by this bit of the US Code). The most relevant text is:


  (b) Permissive Refusal.
  Subject to regulations of the Under Secretary, an air carrier, intrastate air carrier, or foreign air carrier may refuse to transport a passenger or property the carrier decides is, or might be, inimical to safety.


Most air carriers vest the captain with the decision-making authority here: If the captain determines that a passenger may be "inimical to safety" they can refuse to fly the passenger.  There are some restrictions on that authority - notably that the basis for the decision can't be "discriminatory" - which are scattered about in 14CFR (the FARs) and 49USC (Transportation), as well as case law. The FAA has issued at least one legal interpretation discussing this in more detail.



For commercial (or non-commercial) operations under part 91 I wasn't able to find anything specific in 14CFR or 49USC, but in those cases the general consensus seems to be that the PIC has at least the level of authority granted under 49USC44902 - if not broader authority (i.e. if you're going on a personal flight you could refuse to take someone along because you don't like their shoes, and if someone hires you for a commercial flight you could refuse the job if they were acting sketchy).