I have a huge list of IP addresses recorded from various visitors to a website. A huge amount of the visitors, in some months over 70%, came from IP addresses that contained keywords such as google, yahoo, bot, crawler, etc.Does this mean that those users were infact search engine crawlers?If so, why are their so many crawlers in my visitor records in comparison to genuine human visitors? (and if not what's the explanation?)Thanks in advance.UPDATEHere's a few examples of the data:livebot-65-55-209-133.search.live.comcrawl2.cosmixcorp.comcrawl-66-249-70-78.googlebot.comlj511965.crawl.yahoo.netlj611054.inktomisearch.comss125.dal0.gigablast.comcrawl-15.cuill.com
If an visitors IP address contains google or a similar keyword, does this mean they were a crawler?
search engines;analytics;ip address;web crawlers
That looks genuine, as it's fairly hard to spoof a source domain name -- crawlers can generate a lot of traffic. They often check pages for changes quite frequently.You can slow down many of them with a non-standard (but fairly well supported) addition to the Robots Exclusion Protocol -- create a file called robots.txt that's served from your web server's root directory with the following contents:User-agent: *Crawl-delay: 60Where the number on the second line is the number of seconds you want each crawler to wait between page loads on your site. (If you've already got a robots.txt file you'll need to modify it instead.)