At what position or distance from an airport are you considered to be in the "traffic pattern"
This AC describes a fair bit of what/where a traffic pattern is. ultimately it comes down to definition but the answer seems to be what ever the defined traffic pattern says  


  The airport traffic patterns described in part 93 relate primarily to
  those airports where there is a need for unique traffic pattern
  procedures not provided for in part 91. Part 97 addresses instrument
  approach procedures (IAP). At airports without operating control
  towers, part 91 requires only that pilots of airplanes approaching to
  land make all turns to the left, unless light signals or visual
  markings indicate that turns should be made to the right


It seems they are somewhat careful to not explicitly define anything and instead allow some definitions to define it for explicit cases and others to be left to interpretations. The FAA advises that calls be made as far as 10 miles out. 

The FAA technically allows airports to define their traffic pattern themselves:  


  Traffic Pattern Design. Airport owners and operators, in coordination
  with the FAA, are responsible for establishing traffic patterns. The
  FAA encourages airport owners and operators to establish traffic
  patterns as recommended in this AC. Further, left traffic patterns
  should be established, except where obstacles, terrain, and
  noise-sensitive areas dictate otherwise


They do provide some guidance on how to do so in regards to altitude they even allow the pilot to decide 


  Traffic Pattern Altitudes. It is recommended that airplanes observe a
  1,000 foot above ground level (AGL) traffic pattern altitude. Large
  and turbine-powered airplanes should enter the traffic pattern at an
  altitude of 1,500 feet AGL or 500 feet above the established pattern
  altitude. Ultralight aircraft should operate no higher than 500 feet
  below the powered aircraft pattern altitude. A pilot may vary the size
  of the traffic pattern depending on the aircraft’s performance
  characteristics


Going straight to regulations taken from here


  §91.127   Operating on or in the vicinity of an airport in Class E airspace. 
  
  (b) Departures. Each pilot of an aircraft must comply with
  any traffic patterns established for that airport in part 93 of this
  chapter.


and for class D


  §91.129   Operations in Class D airspace. 
  
  (1) Unless required by the applicable distance-from-cloud criteria, each pilot operating a
  large or turbine-powered airplane must enter the traffic pattern at an
  altitude of at least 1,500 feet above the elevation of the airport and
  maintain at least 1,500 feet until further descent is required for a
  safe landing.


and Class C


  §91.130   Operations in Class C airspace. 
  
  (b) Traffic patterns. No person may take off or land an aircraft at a satellite airport within
  a Class C airspace area except in compliance with FAA arrival and
  departure traffic patterns.




If the tower is in operation they can vector you as they see fit (at least here under the FAA) as such they may use pattern terms and organization but I cant tell if the FAA considers it a pattern or not if there is a tower.