<p>What techniques or tools are recommended for finding broken links on a website?</p>

<p>I have access to the logfiles, so could conceivably parse these looking for 404 errors, but would like something automated which will follow (or attempt to follow) all links on a site.</p>

[{'answer_id': 65523, 'author': 'jrudolph', 'author_id': 7647, 'author_profile': 'https://Stackoverflow.com/users/7647', 'pm_score': 6, 'selected': True, 'text': '<p>For Chrome Extension there is <a href="https://chrome.google.com/webstore/detail/hexometer-tool/hikinokpbcadaepfkacpnkhbdjfpfcgc" rel="nofollow noreferrer">hexometer</a></p>\n\n<p>See <a href="http://www.kevinfreitas.net/extensions/linkchecker/" rel="nofollow noreferrer">LinkChecker</a> for Firefox.</p>\n\n<p>For Mac OS there is a tool <a href="http://peacockmedia.co.uk/integrity/" rel="nofollow noreferrer">Integrity</a> which can check URLs for broken links.</p>\n\n<p>For Windows there is <a href="http://home.snafu.de/tilman/xenulink.html" rel="nofollow noreferrer">Xenu\'s Link Sleuth</a>.</p>\n'}, {'answer_id': 65538, 'author': 'Peter Hilton', 'author_id': 2670, 'author_profile': 'https://Stackoverflow.com/users/2670', 'pm_score': 2, 'selected': False, 'text': '<p>Either use a tool that parses your log files and gives you a \'broken links\' report (e.g. <a href="http://www.analog.cx/" rel="nofollow noreferrer">Analog</a> or Google <a href="http://www.google.com/webmasters/tools/" rel="nofollow noreferrer">Webmaster Tools</a>), or run a tool that spiders your web site and reports broken links (e.g. <a href="http://validator.w3.org/checklink" rel="nofollow noreferrer">W3C Link Checker</a>).</p>\n'}, {'answer_id': 65581, 'author': 'Paul Reiners', 'author_id': 7648, 'author_profile': 'https://Stackoverflow.com/users/7648', 'pm_score': 4, 'selected': False, 'text': '<p>I like the <a href="http://validator.w3.org/checklink" rel="noreferrer">W3C Link Checker</a>.</p>\n'}, {'answer_id': 65590, 'author': 'Community', 'author_id': -1, 'author_profile': 'https://Stackoverflow.com/users/-1', 'pm_score': 3, 'selected': False, 'text': '<p>See <a href="http://wummel.github.io/linkchecker/" rel="noreferrer">linkchecker</a> tool:</p>\n\n<blockquote>\n  <p>LinkChecker is a free, GPL licensed website validator. LinkChecker checks links in web documents or full websites.</p>\n</blockquote>\n'}, {'answer_id': 65625, 'author': 'ConroyP', 'author_id': 2287, 'author_profile': 'https://Stackoverflow.com/users/2287', 'pm_score': 0, 'selected': False, 'text': '<p>Your best bet is to knock together your own spider in your scripting language of choice, it could be done recursively along the lines of:</p>\n\n<pre><code>// Pseudo-code to recursively check for broken links\n// logging all errors centrally\nfunction check_links($page)\n{\n    $html = fetch_page($page);\n    if(!$html)\n    {\n        // Log page to failures log\n        ...\n    }\n    else\n    {\n        // Find all html, img, etc links on page\n        $links = find_links_on_page($html);\n        foreach($links as $link)\n        {\n            check_links($link);\n        }\n    }\n}\n</code></pre>\n\n<p>Once your site has gotten a certain level of attention from Google, their <a href="http://www.google.com/webmaster" rel="nofollow noreferrer">webmaster tools</a> are invaluable in showing broken links that users may come across, but this is quite reactionary - the dead links may be around for several weeks before google indexes them and logs the 404 in your webmaster panel.</p>\n\n<p>Writing your own script like above will show you all possible broken links, without having to wait for google (webmaster tool) or your users (404 in access logs) to stumble across them.</p>\n'}, {'answer_id': 68581, 'author': 'scunliffe', 'author_id': 6144, 'author_profile': 'https://Stackoverflow.com/users/6144', 'pm_score': 0, 'selected': False, 'text': '<p>There\'s a windows app called CheckWeb.  Its no longer developed, but it works well, and the code is open (C++ I believe).</p>\n\n<p>You just give it a url, and it will crawl your site (and external links if you choose), reporting any errors, image / page "weight" etc.</p>\n\n<p><a href="http://www.algonet.se/~hubbabub/how-to/checkweben.html" rel="nofollow noreferrer">http://www.algonet.se/~hubbabub/how-to/checkweben.html</a></p>\n'}, {'answer_id': 2396982, 'author': 'Ian Mercer', 'author_id': 224370, 'author_profile': 'https://Stackoverflow.com/users/224370', 'pm_score': 1, 'selected': False, 'text': '<p>In a .NET application you can set IIS to pass all requests to ASP.NET and then in your global error handler you can catch and log 404 errors.  This is something you\'d do in addition to spidering your site to check for internal missing links.  Doing this can help find broken links from OTHER sites and you can then fix them with 301 redirects to the correct page.</p>\n\n<p>To help test your site internally there\'s also the Microsoft SEO toolkit.</p>\n\n<p>Of course the best technique is to avoid the problem at compile time!  In ASP.NET you can get close to this by requiring that all links be generated from static methods on each page so there\'s only ever one location where any given URL is generated.  e.g. <a href="http://www.codeproject.com/KB/aspnet/StronglyTypedPages.aspx" rel="nofollow noreferrer">http://www.codeproject.com/KB/aspnet/StronglyTypedPages.aspx</a></p>\n\n<p>If you want a complete C# crawler, there\'s one here:- <a href="http://blog.abodit.com/2010/03/a-simple-web-crawler-in-c-using-htmlagilitypack/" rel="nofollow noreferrer">http://blog.abodit.com/2010/03/a-simple-web-crawler-in-c-using-htmlagilitypack/</a></p>\n'}, {'answer_id': 7160752, 'author': 'akauppi', 'author_id': 14455, 'author_profile': 'https://Stackoverflow.com/users/14455', 'pm_score': 0, 'selected': False, 'text': '<p><a href="http://www.linktiger.com" rel="nofollow">LinkTiger</a> seems like a very polished (though non-free) service to do this. I\'m not using it, just wanted to add because it was not yet mentioned.</p>\n'}, {'answer_id': 9004718, 'author': 'Jonathan', 'author_id': 654128, 'author_profile': 'https://Stackoverflow.com/users/654128', 'pm_score': 1, 'selected': False, 'text': '<p>Our commercial product <a href="http://www.deeptrawl.com" rel="nofollow">DeepTrawl</a> does this and can be used on both Windows / Mac.</p>\n\n<p>Disclosure: I\'m the lead developer behind DeepTrawl.</p>\n'}, {'answer_id': 15029100, 'author': 'wjbrown', 'author_id': 2100180, 'author_profile': 'https://Stackoverflow.com/users/2100180', 'pm_score': 5, 'selected': False, 'text': '<p>Just found a wget script that does what you are asking for.</p>\n\n<pre><code>wget --spider  -o wget.log  -e robots=off --wait 1 -r -p http://www.example.com\n</code></pre>\n\n<p>Credit for this goes to <a href="http://www.commandlinefu.com/commands/view/8234/check-broken-links-using-wget-as-a-spider" rel="noreferrer">this page</a>.</p>\n'}]
['https://Stackoverflow.com/questions/65515', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/2084/']