1. A system for crawling a site having pages, each page having a reference that identifies the page, comprising: a grouping component that identifies groups of pages with similar content; a pattern component that identifies a reference pattern of a group based on the references of the pages of the group; and a decision component that, after encountering a reference that matches a reference pattern, decides whether to access the page of the encountered reference based on characteristics of the pages of the group of the matching reference pattern.