When departing IFR, when do I switch from Tower to Departure?
When operating within class D airspace (operating Air Traffic Control Tower) pilots are to maintain communication with the tower until instructed otherwise. 

On an IFR flight departing from a tower controlled airport, the tower controller will instruct the pilot when to change frequency to contact the next controller (the IFR Departure Controller).  Remain on the current frequency until instructed to change.

Here is an except from the pertinent regulation (FAR Part 91):


  §91.129   Operations in Class D airspace.
  
  2) Departing flight. Each person—
  
  (i) From the primary airport or satellite airport with an operating control tower must establish and maintain two-way radio communications with the control tower, and thereafter as instructed by ATC while operating in the Class D airspace area; 


Note: It should be noted however, that in the U.S., departing IFR military turboprop/turbojet aircraft (except transport and cargo types) will be instructed to "change to departure control frequency" before takeoff. [ref:FAA JO 7110.65, paragraph 3-9-3 a. 2.]

This requirement allows pilots of single-seat high-performance aircraft, as well as certain other military aircraft, the ability to avoid being distracted by changing to another (e.g., departure control) frequency immediately after takeoff.  Further, many military fighter type aircraft used to have (probably some still do) radios positioned in the cockpit in a location that requires looking down or moving the pilot's head/eyes in such a manner that might cause momentary disorientation, which is not a good idea right after takeoff and close to the ground.

U.S. military control towers and some (perhaps most) joint-use military/civilian control towers have an "override" frequency (same frequency as the departure controller uses), for use in case some emergency situation that occurs just before, during, or immediately after departure requires urgent communication with the pilot (e.g., engine fire on takeoff noted by the tower, or similar type situation).