What determines the direction of rotation for counter-rotating propellers?
Most twin-engine aircraft with counter-rotating propellers have the rotation set up so that the propellers are rotating inward towards the center at the tops of the propeller arc. 



This configuration reduces the P-Factor effect at slow speed high angles of attack, and eliminates the "critical" engine that is present on multi-engine aircraft where both engines rotate in the same direction.

Conventional Multi engine propeller aircraft (both clockwise rotation)


The original XP-38 (X=Experimental) had "inward" counter rotating propellers but only flew a few times before it was destroyed in a crash.
  

There were some tail buffeting problems identified with the XP-38. While trying to solve the tail buffeting problems, various wind tunnel tests were conducted. Some of these tests were done with "outward" rotating propellers. 

These tests showed that "outward" rotating propellers did not reduce tail buffeting BUT they did reduce power on/power off pitching moments, and therefore made for a more stable gun platform. 


  "The P-38 props rotate in the direction they do for the simple reason it reduced the power on/off pitching moment, thus making for a more stable gun platform" - Kelly Johnson, P-38 designer.


The tail buffeting was eventually solved by fitting large leading edge fillets at the wing/fuselage interface. 

The YP-38 (Y=Prototype) and most subsequent versions of the P-38 retained the "outward" counter rotating propellers. This in effect made both engines "critical", but combat performance was more important that low speed performance. 

So while most aircraft with counter rotating propellers have "inward" rotation, a few do have "outward" rotation to gain an operational advantage.