Could a private pilot violate 'commercial' rules if there is no compensation or expense sharing at all?
Yes.  Without compensation, a private pilot can violate 'commercial' rules.  

For starters, the FAA considers the ability to log flight time as compensation.  A legal interpretation to John Harrington issued October 1997 clarifies that point.  The regulation that you are asking about is quoted below.




  § 61.113   Private pilot privileges and limitations: Pilot in command.
  
  (a) Except as provided in paragraphs (b) through (h) of this section, no person who holds a private pilot certificate may act as pilot in command of an aircraft that is carrying passengers or property for compensation or hire; nor may that person, for compensation or hire, act as pilot in command of an aircraft. 




An interpretation to Mike Sommer in part says


  The FAA construes the terms “compensation for hire” very broadly. It does not require a profit, profit motive, or the actual payment of funds. Instead, the FAA views compensation as the receipt of anything of value. In an interpretation letter to John W. Harrington, from Donald Byrne, Assistant Chief Counsel, October 23, 1997, it is stated that, “any reimbursement of expenses (fuel, oil, transportation, lodging, meals, etc.), if conditioned upon the pilot operating the aircraft,’ would constitute compensation.”


In 1985 the FAA chief counsel determined that there had to be a bona-fide common purpose in conducting the flight.  See the Chero Interpretation from December 1985.


  Section 61.118(b) allows a private pilot to share the operating expenses of a flight with his or her passengers. Additionally, the FAA has interpreted 61.118(b) so that the only allowable share-the-costs operations are those which are bona fide, that is, joint ventures for a common purpose with the expenses being defrayed by all passengers and the pilot. Nor does Section 61.118 permit pilots who want to build up time toward their commercial pilot certificates to carry expense sharing passengers to a destination at which they have no particular business. (emphasis added)




To answer your question.


The pilot flies the passenger somewhere they weren't planning on
going, but nonetheless enjoy. I'm assuming this is not a problem,
and happens most days.

There is no problem here.  The pilot would pay for the expenses of the flight anyway so taking a passenger is OK.
The pilot flies the passenger to visit a family member, one they wouldn't have visited otherwise.

This is only legal if the pilot pays for all expenses.  Since the FAA has determined compensation quite broadly I would expect they would also consider food and lodging as compensation as stated previously in a legal interpretation.  The pilot would, for all practical purposes, pretend the passenger was not there and make their own arrangements.
The pilot flies the passenger to a specific recreational event, which they are attending. 

This is legal and the pilot can share the costs with the passenger.
The pilot flies the passenger to a specific recreational event, which only the passenger is attending.

This goes back to #2 above.  The pilot does not have a bona-fide common purpose for making the flight.  They cannot accept any compensation in any form for the flight to be legal.