Papers on fault handling in distributed systems
You might want to have a look at the works that won Tushar D. Chandra, Vassos Hadzilacos, and Sam Toueg the Edsger W. Dijkstra Prize in 2010:


Tushar D. Chandra and Sam Toueg. Unreliable Failure Detectors for Reliable Distributed Systems, Journal of the ACM, 43(2):225-267, 1996
Tushar D. Chandra, Vassos Hadzilacos and Sam Toueg. The Weakest Failure Detector for Solving Consensus, Journal of the ACM, 43(4):685-722, 1996.


These papers introduce the notion of failure detectors in a distributed system in a general and precise framework. Intuitively, they tried to study the minimal amount of failure information needed to solve consensus. It turns out that you don't need a perfect failure detector to solve consensus. Even unreliable failure detectors satisfying certain minimal conditions will suffice for the task. These papers were very influential on how to deal with failures in distributed systems.