What mathmatical model shall be used for describing P2P processes interaction?
If you're looking for a single model that describes all relevant aspects of your system, you'll have a hard time. And building the model will take as much effort as implementing the system itself, if not more. Therefore, break the system down into different aspects, and consider different modelling techniques for each. Examples for aspects might be:


The protocol between two of the peers. This could detect deadlocks or pointless messages in the protocol.
Repeat for every combination of two types of peers.
Behavior of a single peer as it connects with different types of peers.
Could there be problems if a message from peer type X comes in while processing another message from peer type Y?
Behavior of the network as peers come and go. Will it stay connected?
Here, you'd just model "peer connects" and "peer disconnects", rather than the sequence of messages for connecting and disconnecting.
Performance depending on the number of peers of each type.
...