Can an aircraft operator hire a Commercial Pilot to fly their aircraft under part 91?
To answer my own question. As pointed out by @MichaelHall and @pericynthion Tom would not fall under "Commercial operator".

Here is the definition for reference:


  Commercial operator means a person who, for compensation or hire,
  engages in the carriage by aircraft in air commerce of persons or
  property, other than as an air carrier or foreign air carrier or under
  the authority of Part 375 of this title. Where it is doubtful that an
  operation is for “compensation or hire”, the test applied is whether
  the carriage by air is merely incidental to the person's other
  business or is, in itself, a major enterprise for profit.


Another term which comes in useful here is:


  Operational control, with respect to a flight, means the exercise of
  authority over initiating, conducting or terminating a flight.


In this case, Joe would have operational control since it is his aircraft. Since Commercial Operation means "...person who... engages in the carriage by aircraft...", then Joe is the "person" and not the pilot Joe.

Another term within the definition is "air commerce". This means:


  Air commerce means interstate, overseas, or foreign air commerce or
  the transportation of mail by aircraft or any operation or navigation
  of aircraft within the limits of any Federal airway or any operation
  or navigation of aircraft which directly affects, or which may
  endanger safety in, interstate, overseas, or foreign air commerce.


Commerce is the activity of buying or selling. If we look at it from that level, the operator (Joe) is not engaging in air Commerce, because when you are operating an aircraft, you're perfectly allowed to pay for the usage of your own plane. So if this is not air commerce, then it falls under Part 91, and Tom is good to go.

However, if Joe were to engage in air commerce such as shipping stuff, but still hire Joe, then this falls under part 119 then 135 operations.

Refer to this legal interpretation: https://www.faa.gov/about/office_org/headquarters_offices/agc/practice_areas/regulations/interpretations/data/interps/1993/green%20-%20(1993)%20legal%20interpretation.pdf

Based off this interpretation, the aircraft and operator must be separate from the pilot. Also, the operator cannot conduct air commerce flights.