title: How to avoid duplicate entries crawling a site
body: <p>I want to crawl a shop with Ruby, Nokogiri and Mechanize.</p>
  
  <p>On a page with two articles shown, I know that all articles start with <code>.../p/...</code> in the address so that's why I store this in <code>article_links</code>. All <code>/p/</code> links should be displayed.</p>
  
  <p>Normally I would see two addresses: </p>
  
  <pre><code>agent = Mechanize.new
  page = agent.get(exampleshop.com)
  
  article_links = page.links_with(href: %r{.*/p/})
  
  article_links.map do |link|
      article = link.click
      target_URL = page.uri + link.uri #full URL
      puts "#{target_URL}"
  end   
  #crawling stuff on /p/ pages not included here
  </code></pre>
  
  <p>However, in the end each link is duplicated, which already happens before the loop, so I get to see:</p>
  
  <pre><code>exampleshop.com/p/productxy.html
  
  exampleshop.com/p/productxy.html
  
  exampleshop.com/p/productab.html
  
  exampleshop.com/p/productab.html
  </code></pre>
  
  <p>I believe that there are two hrefs with <code>/p/</code> for each product in the site's code. Is there a good way to prevent this? Or is it possible to use Nokogiri CSS in <code>links_with</code>?  </p>