Forcing an honest behavior
Alas, you can't force people to do what the protocol says they should do.

Even well-meaning people who intended to follow the protocol occasionally make mistakes.

There seem to be at least 3 approaches:


crypto theory: assume "good" agents always follow the protocol, while "malicious" agents try to subvert the protocol. Design the crypto protocol such that good agents get what they need, while malicious agents get nothing.
game theory: assume every agent looks out only for his own individual interest. Use mechanism design to maximize the total benefit to everyone.
distributed fault-tolerant network: assume every agent makes an occasional mistake, and a few 'bot agents spew out many maliciously-crafted messages. Detect and isolate the 'bot nodes until they are fixed; use error detection and correction (EDAC) to fix the occasional mistake; use convergent protocols that eventually settle into a useful state no matter what initial mis-information is stored in the routing tables.


mechanism design
In game theory, designing a situation (such as setting up the rules of an auction) such that people who are selfishly looking out only for their own individual interests end up doing what the designer wants them to do is called "mechanism design".
In particular, using implementation theory, situations can be designed such that the final outcome maximizes the total benefit to everyone, avoiding poorly-designed situations such as the "tragedy of the commons" or "prisoner's dilemma" where things happen that are not in anyone's long-term interest.

Some of the most robust such processes are designed to be incentive compatible.

The game theory typically makes the simplifying assumption that all relevant agents are "rational".
In game theory, "rational" means that an agent prefers some outcomes to other outcomes, is willing and able to change his actions in a way that he expects (given the information available to him) will result in a more preferred outcome (his own narrow self-interest), and he is smart enough to realize that other rational agents will act similarly to try to obtain the outcome that is most preferred out of all the possible outcomes that might result from that choice of action.

A designer may temporarily make the simplifying assumption that all people only act according to their own narrow self-interest.
That assumption makes it easier to design a situation using implementation theory.
However, after the design is finished,
it doesn't matter whether people act according to their own narrow self-interest ( "Homo economicus" ),
or whether they are altruistic and want to maximize the total net benefit to everyone -- in a properly designed situation, both kinds of people make exactly the same choices and the final outcome maximizes the total benefit to everyone.

convergent protocols

When designing a routing protocol,
each node in the network sends messages to its neighbors passing on information about what other nodes are reachable from that node.

Alas, occasionally these messages have errors.
Worse, sometimes a node is mis-configured and spews out many misleading and perhaps even maliciously-crafted messages.

Even though we humans know the message might be incorrect, we typically design the protocol so that a properly-functioning node trusts every message and stores the information in its routing table, and makes its decisions as if it believes that information to be entirely true.

Once some human turns off a bad node (or disconnects it from the network), we typically design the protocol to rapidly pass good information to flush out the corrupt information, and so quickly converge on a useful state.

combined approaches

Algorithmic mechanism design
seems to try to combine the fault-tolerant network approach and the game-theory mechanism approach.

@Yoichi Hirai and Aaron, thank you for pointing out some interesting attempts to combine game theory and cryptography.