When may pilots leave the flight deck during flight?
14 CFR 121.543


  (a) Except as provided in paragraph (b) of this section, each required flight crewmember on flight deck duty must remain at the assigned duty station with seat belt fastened while the aircraft is taking off or landing, and while it is en route.
  
  (b) A required flight crewmember may leave the assigned duty station -
  
  
    (1) If the crewmember's absence is necessary for the performance of duties in connection with the operation of the aircraft;
    
    (2) If the crewmember's absence is in connection with physiological needs; or
    
    (3) If the crewmember is taking a rest period, and relief is provided -
    
    
      (i) In the case of the assigned pilot in command during the en route cruise portion of the flight, by a pilot who holds an airline transport pilot certificate not subject to the limitations in § 61.167 of this chapter and an appropriate type rating, is currently qualified as pilot in command or second in command, and is qualified as pilot in command of that aircraft during the en route cruise portion of the flight. A second in command qualified to act as a pilot in command en route need not have completed the following pilot in command requirements: The 6-month recurrent flight training required by § 121.433(c)(1)(iii); the operating experience required by § 121.434; the takeoffs and landings required by § 121.439; the line check required by § 121.440; and the 6-month proficiency check or simulator training required by § 121.441(a)(1); and
      
      (ii) In the case of the assigned second in command, by a pilot qualified to act as second in command of that aircraft during en route operations. However, the relief pilot need not meet the recent experience requirements of § 121.439(b).
    
  


tl;dr:
You need at least one pilot at the controls at all times, unless completely necessary to operate the aircraft, or take care of physiological needs.