how to make forest with different types of trees
Mask system's density and Render Groups

To avoid interaction between two particle system you should somehow mask their density in a complementary way, so where one is the other isn't.



Let's start from creating a simple forest with two type of trees and only one particle system:

After modeling the tree types, you should assigning your sample trees objects to a group (select and Ctrl+G):



Then add to your surface a particle system with these two characteristics:


Hair particles initial orientation on Global Z with random phase as well described in this answer to get trees pointin straight up and with some random rotation around the Z axis
Render option set to Group (pointing to the previously created group)
with Rotation (rotate your object towart the X axis as in the picture) and Pick random




The last mentioned option will pick a random object from the target group, resulting in an assorted forest.

To get full control, you can use the Count option instead which would let you choose in wich proportion each tree type is present in the forest



Above you can see a fores made by 90% of pines and 10% of Betullae.



Now add a second particle system, this time made by 20% Pines and 80% Betullaes:



Go to weight paint, create a new vertex group layers and start paining red the places where the Pines (90%) will be. Then duplicate the vertex group (Copy vertex grup from the menue), Invert the weight, paint some 0 weight areas (where you don't want anty tree) and rename the vertex group to Pines 20%.





Assign to each particle system the appropriate vertex weight group and you'll have a two-tree-type forest:



Apart from vertex weight, you can mask the two particles system even with textures, or splitting the surface in many different objects... it depends on what your project's workflow is.



To avoid intersection between particles you should keep an eye on this related question. Sadly there is not a exaustive answer yet.