I return 403 Forbidden for all URLs, including the robots.txt
file. Why is the site still being crawled?
我对所有网址（包括 robots.txt 文件）都返回了 403 Forbidden。为什么网站仍在被抓取？
The 403 Forbidden HTTP status code, as well as other 4xx
HTTP status codes, is interpreted as the robots.txt file doesn't exist. This means
that crawlers will generally assume that they can crawl all URLs of the website. In
order to block crawling of the website, the robots.txt must be returned with a
200 OK HTTP status code, and must contain an appropriate
disallow rule.
403 Forbidden HTTP 状态代码以及其他 4xx HTTP 状态代码代表 robots.txt 文件不存在。这意味着抓取工具一般会假定可以抓取该网站的所有网址。若要禁止抓取网站，您必须针对 robots.txt 返回 200 OK HTTP 状态代码，其中必须包含相应的 disallow 规则。