title: How can scrapy be used to extract the link graph of a website?
body: <p>Given a starting URL <code>start</code> (and some rules on admissible domains etc) I would like to produce a directed graph (V, E) where the nodes in <code>V</code> are the pages reachable from <code>start</code>, and there is an arc <code>(u,v)</code> in <code>E</code> whenever there is a hyperlink on page <code>u</code> pointing to page <code>v</code>. </p>
  
  <p>Is there a simple way to obtain such a graph with <code>scrapy</code>? I would also be happy using another open source tool if it can achieve the goal more easily/nicely.</p>