title: Automated link-checker for system testing
body: <p>I often have to work with fragile legacy websites that break in unexpected ways when logic or configuration are updated.</p>
  
  <p>I don't have the time or knowledge of the system needed to create a <a href="http://selenium.org/" rel="noreferrer">Selenium</a> script. Besides, I don't want to check a specific use case - I want to verify every link and page on the site. </p>
  
  <p>I would like to create an automated system test that will spider through a site and check for broken links and crashes. Ideally, there would be a tool that I could use to achieve this. It should have as many as possible of the following features, in descending order of priority:</p>
  
  <ul>
  <li>Triggered via script</li>
  <li>Does not require human interaction</li>
  <li>Follows all links including anchor tags and links to CSS and js files</li>
  <li>Produces a log of all found 404s, 500s etc.</li>
  <li>Can be deployed locally to check sites on intranets</li>
  <li>Supports cookie/form-based authentication</li>
  <li>Free/Open source</li>
  </ul>
  
  <p>There are many partial solutions out there, like <a href="http://fitnesse.org/" rel="noreferrer">FitNesse</a>, <a href="https://addons.mozilla.org/en-US/firefox/addon/532" rel="noreferrer">Firefox's LinkChecker</a> and <a href="http://validator.w3.org/checklink" rel="noreferrer">the W3C link checker</a>, but none of them do everything I need.</p>
  
  <p>I would like to use this test with projects using a range of technologies and platforms, so the more portable the solution the better.</p>
  
  <p>I realise this is no substitute for proper system testing, but it would be very useful if I had a convenient and automatable way of verifying that no part of the site was obviously broken.</p>