九个用来构建容错系统的开源工具
容错

> 
> 这些开源工具可以最大化延长运行时间并且在最大程度上减少问题。
> 
> 
> 


![](/data/attachment/album/202008/30/205036eqh1j8hhss9skf57.jpg)


我一直对 Web 开发和软件体系结构很感兴趣，因为我喜欢看到一个工作系统的宏观视图。无论是构建一个移动应用程序还是一个 Web 应用程序，都必须连接到互联网，在不同的模块中交换数据，这意味着你需要 Web 服务。


如果选择云系统作为应用程序的后端，则可以利用更强大的计算能力，因为后端服务将会在水平和垂直方向上进行扩展并编排不同的服务。但无论你是否使用云后端，建造一个灵活、稳定、快速又安全的容错系统是必不可少的。


要了解容错系统，让我们以脸书、亚马逊、谷歌和奈飞为例。数以亿计的用户会同时接入这些平台并通过对等网络和用户-服务器网络传输大量数据，你可以肯定这其中还存在许多的带有不法目的的恶意用户，例如黑客攻击和拒绝服务（DoS）攻击。即使如此，这些平台无需停机也可以全年无休地运转。


虽然机器学习和智能算法是这些系统的基础，但它们实现持续的服务而不停机一分钟的事实值得称赞。它们昂贵的硬件设备和巨大的数据中心当然十分重要，但是支持服务的精密软件设计也同样重要。而且容错系统是一个构建如此精密系统的法则之一。


### 生产过程中导致错误的两种行为


这是考虑容错系统的另一种方法。当你在本地运行应用程序服务时，每件事似乎都很完美。棒极了！但当你提升服务到生产环境时，一切都会变得一团糟。在这种情况下，容错系统通过解决两个问题来提供帮助：故障停止行为和拜占庭行为。


#### 故障停止行为


故障停止行为是运行中系统突然停止运行或者系统中的某些部分发生了故障。服务器停机时间和数据库不可访问都属于此种类型。举个例子，在下图中，由于服务 2 无法访问，因此服务 1 无法与服务 2 进行通信。


![服务 2 停机导致的故障停止行为](/data/attachment/album/202008/30/205112y5ylcyculncqq7o3.jpg "Fail-stop behavior due to Service 2 downtime")


但是，如果服务之间存在网络问题，也会出现此问题，如下图所示：


![网络故障导致的故障停止行为](/data/attachment/album/202008/30/205113z4ykg4grysegkob3.jpg "Fail-stop behavior due to network failure")


#### 拜占庭行为


拜占庭行为是指系统在持续运行，但并没有产生预期行为（例如：错误的数据或者无效的数据）。


如果服务 2 的数据（值）已损坏则可能会发生拜占庭故障，即使服务看起来运行得很好，比如下面的例子:


![因服务损坏而导致的拜占庭故障](/data/attachment/album/202008/30/205113qg96xllkqkxk7xcz.jpg "Byzantine failure due to corrupted service")


或者，可能存在恶意的中间人在服务之间进行拦截，并注入了不需要的数据：


![恶意中间人导致的拜占庭故障](/data/attachment/album/202008/30/205117gnzppmt6p7iq4lzu.jpg "Byzantine failure due to malicious middleman")


无论是故障停止和拜占庭行为，都不是理想的情况，因此我们需要一些预防或修复它们的手段。这里容错系统就起作用了。以下是可以帮助你解决这些问题的 8 个开源工具。


### 构建容错系统的工具


尽管构建一个真正实用的容错系统涉及到深入的“分布式计算理论”和复杂的计算机科学原理，但有许多的软件工具（其中许多是开源软件）通过构建容错系统来减轻不良后果的影响。


#### 断路模式：Hystrix 和 Resilience4j


[断路模式](https://martinfowler.com/bliki/CircuitBreaker.html)是一种技术，它有助于在服务失败时返回准备好的虚拟回应或者简单回应。


![断路模式](/data/attachment/album/202008/30/205120uiw947fgfula94pa.jpg "Circuit breaker pattern")


奈飞开源的 [Hystrix](https://github.com/Netflix/Hystrix/wiki) 是断路模式中最流行的应用。


我之前工作过的很多家公司都在用这款出色的工具。令人意外的是，奈飞宣布将不再更新 Hystrix（是的，我知道了）。相反，奈飞建议使用另一种支持 Java 8 和函数式编程的 [Resilence4j](https://github.com/resilience4j/resilience4j) 之类的替代解决方案，或者类似于 [Adaptive Concurrency Limit](https://medium.com/@NetflixTechBlog/performance-under-load-3e6fa9a60581) 的替代解决方案。


#### 负载均衡：Nginx 和 HaProxy


负载均衡是分布式系统中最基本的概念之一，要想拥有一个生产质量的环境，必须有负载均衡的存在。要理解负载均衡器，首先我们需要明白冗余的概念。每个生产级的 Web 服务都有多个服务器在某个服务器宕机时提供冗余来接管和维持服务。


![负载均衡](/data/attachment/album/202008/30/205124jv01giikl55v58yi.jpg "Load balancer")


想想现代飞机：它们的双引擎提供冗余，使它们即使在一个引擎着火的情况下也能安全的着陆。（这也有助于大多数商用飞机拥有最先进的自动化系统）。但是，拥有多引擎（或者多服务器）也意味着必须存在一些调度机制在故障发生时有效地对系统进行路由。


负载均衡器是一种通过平衡多个服务节点来优化大流量事务的设备或者软件。举个例子，当数以千计的请求涌入时，负载均衡器可以作为中间层在不同的服务器间进行路由和平均分配流量。如果一台服务器宕机，负载均衡器会将请求转发给其它运行良好的服务器。


有许多可用的负载均衡器，但其中最出名的两个就是 Nginx 和 HaProxy。


[Nginx](https://www.nginx.com) 不仅仅是一个负载均衡器，它还是 HTTP 和反向代理服务器、邮件代理服务器和通用 TCP/UDP 代理服务器。Groupon、Capital One、Adobe 和 NASA 等公司都在使用它。


[HaProxy](https://www.haproxy.org) 也很受欢迎，因为它是一个免费的、非常快且可靠的解决方案，它为基于 TCP 和 HTTP 的应用程序提供高可用性、负载平衡和代理。许多大型网络公司，包括 Github、Reddit、Twitter 和 Stack Overflow 都使用 HaProxy。是的，Red Hat Enterprise Linux 同样支持 HaProxy 设置。


#### 参与者模型：Akka


[参与者模型](https://en.wikipedia.org/wiki/Actor_model)是一种并发设计模式，当作为基本计算单位的“参与者”接收到消息时，它会分派责任。一个参与者可以创建更多的参与者，并将消息委派给他们。


[Akka](https://akka.io) 是最著名的参与者模型实现之一。该框架同时支持基于 JVM 的 Java 和 Scala。


#### 使用消息队列的异步、非阻塞 I/O：Kafka 和 RabbitMQ


多线程开发在过去很流行，但是现在已经不鼓励这种做法了，取而代之的是异步的、非阻塞的 I/O 模式。对于 Java，这一点在 [EnterpriseJavaBean（EJB）规范](https://jcp.org/aboutJava/communityprocess/final/jsr220/index.html)中得到了明确的规定：



> 
> “企业 bean 一定不能使用线程同步原语来同步多个实例的执行。”
> 
> 
> “企业 bean 不得试图去管理线程。企业 bean 不得试图启动、停止、挂起或恢复线程，或者去更改线程的优先级或者名称。企业 bean 不得试图管理线程组。”
> 
> 
> 


如今，虽然还有其他做法，如流 API 和参与者模型，但像 [Kafka](https://kafka.apache.org) 和[RabbitMQ](https://www.rabbitmq.com) 之类的消息队列为异步和非阻塞 I/O 功能提供了开箱即用的支持，同时它们也是功能强大的开源工具，通过处理并发进程可以替代线程。


#### 其他的选择：Eureka 和 Chaos Monkey


用于容错系统其它有用的工具包括奈飞的 [Eureka](https://github.com/Netflix/eureka) 之类的监控工具，以及像 [Chaos Monkey](https://github.com/Netflix/chaosmonkey) 这样的压力测试工具。它们旨在通过在较低环境中的测试，如集成（INT）、质量保障（QA）和用户接受测试（UAT）来早早发现潜在问题以防止在转移到生产环境之前出现潜在问题。


你在使用什么开源工具来构建一个容错系统呢？请在评论中分享你的最爱。




---


via: <https://opensource.com/article/19/3/tools-fault-tolerant-system>


作者：[Bryant Son](https://opensource.com/users/brson) 选题：[lujun9972](https://github.com/lujun9972) 译者：[chenmu-kk](https://github.com/chenmu-kk) 校对：[wxy](https://github.com/wxy)


本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出


OK
I've always been interested in web development and software architecture because I like to see the broader picture of a working system. Whether you are building a mobile app or a web application, it has to be connected to the internet to exchange data among different modules, which means you need a web service.

If you use a cloud system as your application's backend, you can take advantage of greater computing power, as the backend service will scale horizontally and vertically and orchestrate different services. But whether or not you use a cloud backend, it's important to build a *fault-tolerant system*—one that is resilient, stable, fast, and safe.

To understand fault-tolerant systems, let's use Facebook, Amazon, Google, and Netflix as examples. Millions and billions of users access these platforms simultaneously while transmitting enormous amounts of data via peer-to-peer and user-to-server networks, and you can be sure there are also malicious users with bad intentions, like hacking or denial-of-service (DoS) attacks. Even so, these platforms can operate 24 hours a day and 365 days a year without downtime.

Although machine learning and smart algorithms are the backbones of these systems, the fact that they achieve consistent service without a single minute of downtime is praiseworthy. Their expensive hardware and gigantic datacenters certainly matter, but the elegant software designs supporting the services are equally important. And the fault-tolerant system is one of the principles to build such an elegant system.

## Two behaviors that cause problems in production

Here's another way to think of a fault-tolerant system. When you run your application service locally, everything seems to be fine. Great! But when you promote your service to the production environment, all hell breaks loose. In a situation like this, a fault-tolerant system helps by addressing two problems: Fail-stop behavior and Byzantine behavior.

### Fail-stop behavior

Fail-stop behavior is when a running system suddenly halts or a few parts of the system fail. Server downtime and database inaccessibility fall under this category. For example, in the diagram below, Service 1 can't communicate with Service 2 because Service 2 is inaccessible:

![Fail-stop behavior due to Service 2 downtime Fail-stop behavior due to Service 2 downtime](https://opensource.com/sites/default/files/uploads/1_errordowntimeservice.jpg)

Fail-stop behavior due to Service 2 downtime

But the problem can also occur if there is a network problem between the services, like this:

![Fail-stop behavior due to network failure Fail-stop behavior due to network failure](https://opensource.com/sites/default/files/uploads/2_errordowntimenetwork.jpg)

Fail-stop behavior due to network failure

### Byzantine behavior

Byzantine behavior is when the system continuously runs but doesn't produce the expected behavior (e.g., wrong data or an invalid value).

Byzantine failure can happen if Service 2 has corrupted data or values, even though the service looks to be operating just fine, like in this example:

![Byzantine failure due to corrupted service Byzantine failure due to corrupted service](https://opensource.com/sites/default/files/uploads/3_byzantinefailuremalicious.jpg)

Byzantine failure due to corrupted service

Or, there can be a malicious middleman intercepting between the services and injecting unwanted data:

![Byzantine failure due to malicious middleman Byzantine failure due to malicious middleman](https://opensource.com/sites/default/files/uploads/4_byzantinefailuremiddleman.jpg)

Byzantine failure due to malicious middleman

Neither fail-stop nor Byzantine behavior is a desired situation, so we need ways to prevent or fix them. That's where fault-tolerant systems come into play. Following are eight open source tools that can help you address these problems.

## Tools for building a fault-tolerant system

Although building a truly practical fault-tolerant system touches upon in-depth *distributed computing theory* and complex computer science principles, there are many software tools—many of them, like the following, open source—to alleviate undesirable results by building a fault-tolerant system.

### Circuit-breaker pattern: Hystrix and Resilience4j

The [circuit-breaker pattern](https://martinfowler.com/bliki/CircuitBreaker.html) is a technique that helps to return a prepared dummy response or a simple response when a service fails:

![Circuit breaker pattern Circuit breaker pattern](https://opensource.com/sites/default/files/uploads/5_circuitbreakerpattern.jpg)

Circuit breaker pattern

Netflix's open source ** Hystrix** is the most popular implementation of the circuit-breaker pattern.

Many companies where I've worked previously are leveraging this wonderful tool. Surprisingly, Netflix announced that it will no longer update Hystrix. (Yeah, I know.) Instead, Netflix recommends using an alternative solution like [ Resilence4j](https://github.com/resilience4j/resilience4j), which supports Java 8 and functional programming, or an alternative practice like

[Adaptive Concurrency Limit](https://medium.com/@NetflixTechBlog/performance-under-load-3e6fa9a60581).

### Load balancing: Nginx and HaProxy

Load balancing is one of the most fundamental concepts in a distributed system and must be present to have a production-quality environment. To understand load balancers, we first need to understand the concept of *redundancy*. Every production-quality web service has multiple servers that provide redundancy to take over and maintain services when servers go down.

![Load balancer Load balancer](https://opensource.com/sites/default/files/uploads/7_loadbalancer.jpg)

Think about modern airplanes: their dual engines provide redundancy that allows them to land safely even if an engine catches fire. (It also helps that most commercial airplanes have state-of-art, automated systems.) But, having multiple engines (or servers) means that there must be some kind of scheduling mechanism to effectively route the system when something fails.

A load balancer is a device or software that optimizes heavy traffic transactions by balancing multiple server nodes. For instance, when thousands of requests come in, the load balancer acts as the middle layer to route and evenly distribute traffic across different servers. If a server goes down, the load balancer forwards requests to the other servers that are running well.

There are many load balancers available, but the two best-known ones are Nginx and HaProxy.

[ Nginx](https://www.nginx.com) is more than a load balancer. It is an HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server. Companies like Groupon, Capital One, Adobe, and NASA use it.

[ HaProxy](https://www.haproxy.org) is also popular, as it is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. Many large internet companies, including GitHub, Reddit, Twitter, and Stack Overflow, use HaProxy. Oh and yes, Red Hat Enterprise Linux also supports HaProxy configuration.

### Actor model: Akka

The [actor model](https://en.wikipedia.org/wiki/Actor_model) is a concurrency design pattern that delegates responsibility when an *actor*, which is a primitive unit of computation, receives a message. An actor can create even more actors and delegate the message to them.

[ Akka](https://akka.io) is one of the most well-known tools for the actor model implementation. The framework supports Java and Scala, which are both based on JVM.

### Asynchronous, non-blocking I/O using messaging queue: Kafka and RabbitMQ

Multi-threaded development has been popular in the past, but this practice has been discouraged and replaced with asynchronous, non-blocking I/O patterns. For Java, this is explicitly stated in its [Enterprise Java Bean (EJB) specifications](https://jcp.org/aboutJava/communityprocess/final/jsr220/index.html):

"An enterprise bean must not use thread synchronization primitives to synchronize execution of multiple instances.

"The enterprise bean must not attempt to manage threads. The enterprise bean must not attempt to start, stop, suspend, or resume a thread, or to change a thread's priority or name. The enterprise bean must not attempt to manage thread groups."


Now, there are other practices like stream APIs and actor models. But messaging queues like [ Kafka](https://kafka.apache.org) and

[offer the out-of-box support for asynchronous and non-blocking IO features, and they are powerful open source tools that can be replacements for threads by handling concurrent processes.](https://www.rabbitmq.com)

**RabbitMQ**### Other options: Eureka and Chaos Monkey

Other useful tools for fault-tolerant systems include monitoring tools, such as Netflix's ** Eureka**, and stress-testing tools, like

**. They aim to discover potential issues earlier by testing in lower environments, like integration (INT), quality assurance (QA), and user acceptance testing (UAT), to prevent potential problems before moving to the production environment.**

[Chaos Monkey](https://github.com/Netflix/chaosmonkey)What open source tools are you using for building a fault-tolerant system? Please share your favorites in the comments.

## 6 Comments