I recently analysed the week-on-week crawl request made by google bot to my server. Here is the graph for the same.Duplicate requests are when more than 1 request was made by google-bot for the same page within a week.I am concerned about the amount of duplicate requests that the bot is making. This results in a significant waste of its bandwidth for crawling my website. On analysing the pages being requested by bot, I could see that a lot of requests were made for pages with low value(less content, lower SERP). Such pages are like list results for certain combination of filters in which majority of the combinations do not offer much value.So, I am pondering whether it is wise to block the google-bot from accessing such low-value pages and concentrate on high value sitemap pages.I had already removed such pages from my sitemap.
Block Google from crawling low value pages
seo;web crawlers;googlebot;crawl rate