如何监测微型的网站服务
监控
![](/data/attachment/album/202207/18/105829gzviausw5wg7wwxb.jpg)


你好! 我最近又开始运行一些服务器（[nginx playground](https://nginx-playground.wizardzines.com)、[mess with dns](https://messwithdns.net)、[dns lookup](https://dns-lookup.jvns.ca)），所以我一直在考虑监控问题。


最初我并不完全清楚如何监控这些网站，所以我想快速写下我是如何做到的。


我根本不打算谈如何监控大型的、严肃的关键任务网站，只谈微型的不重要的网站。


### 目标：在操作上几乎不花时间


我希望网站大部分时间都能正常工作，但我也希望不用在持续的运营上花费时间。


我最初对运行服务器非常警惕，因为在我的上一份工作中，我是 24/7 轮流值班，负责一些关键的服务，在我的印象中，“负责服务器”意味着“在凌晨 2 点被叫起来修理服务器”和“有很多复杂的仪表盘”。


所以有一段时间我只做静态网站，这样我就不用考虑服务器的问题。


但最终我意识到，我所要写的任何服务器的风险都很低，如果它们偶尔宕机 2 小时也没什么大不了的，我只需设置一些非常简单的监控来帮助它们保持运行。


### 没有监控很糟糕


起初，我根本没有为我的服务器设置任何监控。这样做的结果是非常可预见的：有时网站坏了，而我却没有发现，直到有人告诉我！


### 步骤 1：uptime 检查器


第一步是建立一个 uptime 检查器。外面有很多这样的东西，我现在使用的是 [updown.io](https://updown.io/) 和 [uptime robot](https://uptimerobot.com/)。我更喜欢 updown 的用户界面和 [定价](https://updown.io/#pricing) 结构（它是按请求而不是按月收费），但 uptime 机器人有一个更慷慨的免费套餐。


它们会：


1. 检查网站是否正常
2. 如果出现故障，它会给我发电子邮件


我发现电子邮件通知对我来说是一个很好的通知级别，如果网站宕机，我会很快发现，但它不会吵醒我或做其它的什么打扰。


### 步骤 2：端到端的健康检查


接下来，让我们谈谈“检查网站是否正常”到底是什么意思。


起初，我只是把我的健康检查端点之一变成一个函数，无论如何都会返回 `200 OK`。


这倒是挺有用的 – 它告诉我服务器是启动着的！


但不出所料，我遇到了问题，因为它没有检查 API 是否真的在 *工作* – 有时健康检查成功了，尽管服务的其他部分实际上已经进入了一个糟糕的状态。


所以我更新了它，让它真正地发出 API 请求，并确保它成功了。


我所有的服务都只做了很少的事情（nginx playground 只有一个端点），所以设置一个健康检查是非常容易的，它实际上贯穿了服务应该做的大部分动作。


下面是 nginx playground 的端到端健康检查处理程序的样子。它非常基本：它只是发出一个 POST 请求（给自己），并检查该请求是成功还是失败。



```
    func healthHandler(w http.ResponseWriter, r *http.Request) {
        // make a request to localhost:8080 with `healthcheckJSON` as the body
        // if it works, return 200
        // if it doesn't, return 500
        client := http.Client{}
        resp, err := client.Post("http://localhost:8080/", "application/json", strings.NewReader(healthcheckJSON))
        if err != nil {
            log.Println(err)
            w.WriteHeader(http.StatusInternalServerError)
            return
        }
        if resp.StatusCode != http.StatusOK {
            log.Println(resp.StatusCode)
            w.WriteHeader(http.StatusInternalServerError)
            return
        }
        w.WriteHeader(http.StatusOK)
    }


```

### 健康检查频率：每小时一次


现在，我大部分健康检查每小时运行一次，有些每 30 分钟运行一次。


我每小时运行一次，因为 [updown.io](http://updown.io) 的定价是按健康检查次数计算的，我正在监控 18 个不同的 URL，而且我想把我的健康检查预算保持在 5 美元/年的最低水平。


花一个小时来发现这些网站中的一个出现故障，对我来说是可以的 – 如果有问题，我也不能保证能很快修复它。


如果可以更频繁地运行它们，我可能会每 5-10 分钟运行一次。


### 步骤 3：第三步：如果健康检查失败，自动重新启动


我的一些网站在 [fly.io](http://fly.io) 上，fly 有一个相当标准的功能，我可以为一个服务配置一个 HTTP 健康检查，如果健康检查失败，就重新启动服务。


“经常重启”是一个非常有用的策略来弥补我尚未修复的 bug，有一段时间，nginx playground 有一个进程泄漏，`nginx` 进程没有被终止，所以服务器的内存一直在耗尽。


通过健康检查，其结果是，每隔一天左右就会发生这样的情况：


* 服务器的内存用完了
* 健康检查开始失败
* 它被重新启动
* 一切又正常了
* 几个小时后再次重复整个传奇


最终，我开始实际修复进程泄漏，但很高兴有一个解决方法可以在我拖延修复 bug 时保持运行。


这些用于决定是否重新启动服务的运行状况检查更频繁地运行：每 5 分钟左右。


### 这不是监控大型服务的最佳方式


这可能很明显，我在一开始就已经说过了，但是“编写一个 HTTP 健康检查”并不是监控大型复杂服务的最佳方法。 但我不会深入讨论，因为这不是这篇文章的主题。


### 到目前为止一直运行良好！


我最初在 3 个月前的四月写了这篇文章，但我一直等到现在才发布它以确保整个设置正常工作。


这带来了很大的不同 – 在我遇到一些非常愚蠢的停机问题之前，现在在过去的几个月里，网站的运行时间达到了 99.95%！




---


via: <https://jvns.ca/blog/2022/07/09/monitoring-small-web-services/>


作者：[Julia Evans](https://jvns.ca/) 选题：[lujun9972](https://github.com/lujun9972) 译者：[geekpi](https://github.com/geekpi) 校对：[wxy](https://github.com/wxy)


本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出


OK
Hello! I’ve started to run a few more servers recently
([nginx playground](https://nginx-playground.wizardzines.com),
[mess with dns](https://messwithdns.net),
[dns lookup](https://dns-lookup.jvns.ca)), so I’ve been
thinking about monitoring.

It wasn’t initially totally obvious to me how to monitor these websites, so I wanted to quickly write up what how I did it.

I’m not going to talk about how to monitor Big Serious Mission Critical websites at all, only tiny unimportant websites.

### goal: spend approximately 0 time on operations

I want the sites to mostly work, but I also want to spend approximately 0% of my time on the ongoing operations.

I was initially very wary of running servers at all because at my last job I was on a 24/7 oncall rotation for some critical services, and in my mind “being responsible for servers” meant “get woken up at 2am to fix the servers” and “have lots of complicated dashboards”.

So for a while I only made static websites so that I wouldn’t have to think about servers.

But eventually I realized that any server I was going to write was going to be very low stakes, if they occasionally go down for 2 hours it’s no big deal, and I could just set up some very simple monitoring to help keep them running.

### not having monitoring sucks

At first I didn’t set up any monitoring for my servers at all. This had the extremely predictable outcome of – sometimes the site broke, and I didn’t find out about it until somebody told me!

### step 1: an uptime checker

The first step was to set up an uptime checker. There are tons of these out
there, the ones I’m using right now are [updown.io](https://updown.io/) and
[uptime robot](https://uptimerobot.com/). I like updown’s user interface and
[pricing](https://updown.io/#pricing) structure more (it’s per request instead of a monthly fee), but uptime
robot has a more generous free tier.

These

- check that the site is up
- if it goes down, it emails me

I find that email notifications are a good level for me, I’ll find out pretty quickly if the site goes down but it doesn’t wake me up or anything.

### step 2: an end-to-end healthcheck

Next, let’s talk about what “check that the site is up” actually means.

At first I just made one of my healthcheck endpoints a function that returned
`200 OK`

no matter what.

This is kind of useful – it told me that the server was on!

But unsurprisingly I ran into problems because it wasn’t checking that the API
was actually *working* – sometimes the healthcheck succeeded even though the
rest of the service had actually gotten into a bad state.

So I updated it to actually make a real API request and make sure it succeeded.

All of my services do very few things (the nginx playground has just 1 endpoint), so it’s pretty easy to set up a healthcheck that actually runs through most of the actions the service is supposed to do.

Here’s what the end-to-end healthcheck handler for the nginx playground looks like. It’s very basic: it just makes another POST request (to itself) and checks if that request succeeds or fails.

```
func healthHandler(w http.ResponseWriter, r *http.Request) {
// make a request to localhost:8080 with `healthcheckJSON` as the body
// if it works, return 200
// if it doesn't, return 500
client := http.Client{}
resp, err := client.Post("http://localhost:8080/", "application/json", strings.NewReader(healthcheckJSON))
if err != nil {
log.Println(err)
w.WriteHeader(http.StatusInternalServerError)
return
}
if resp.StatusCode != http.StatusOK {
log.Println(resp.StatusCode)
w.WriteHeader(http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusOK)
}
```


### healthcheck frequency: hourly

Right now I’m running most of my healthchecks every hour, and some every 30 minutes.

I run them hourly because updown.io’s pricing is per healthcheck, I’m monitoring 18 different URLs, and I wanted to keep my healthcheck budget pretty minimal at $5/year.

Taking an hour to find out that one of these websites has gone down seems ok to me – if there is a problem there’s no guarantee I’ll get to fixing it all that quickly anyway.

If it were free to run them more often I’d probably run them every 5-10 minutes instead.

### step 3: automatically restart if the healthcheck fails

Some of my websites are on fly.io, and fly has a pretty standard feature where I can configure a HTTP healthcheck for a service and restart the service if the healthcheck starts failing.

“Restart a lot” is a very useful strategy to paper over bugs that I haven’t
gotten around to fixing yet – for a while the nginx playground had a process
leak where `nginx`

processes weren’t getting terminated, so the server kept
running out of RAM.

With the healthcheck, the result of this was that every day or so, this would happen:

- the server ran out of RAM
- the healthcheck started failing
- it get restarted
- everything was fine again
- repeat the whole saga again some number of hours later

Eventually I got around to actually fixing the process leak, but it was nice to have a workaround in place that could keep things running while I was procrastinating fixing the bug.

These healthchecks to decide whether to restart the service run more often: every 5 minutes or so.

### this is not the best way to monitor Big Services

This is probably obvious and I said this already at the beginning, but “write one HTTP healthcheck” is not the best approach for monitoring a large complex service. But I won’t go into that because that’s not what this post is about.

### it’s been working well so far!

I originally wrote this post 3 months ago in April, but I waited until now to publish it to make sure that the whole setup was working.

It’s made a pretty big difference – before I was having some very silly downtime problems, and now for the last few months the sites have been up 99.95% of the time!