title: Spider a Website and Return URLs Only
body: <p>I'm looking for a way to pseudo-spider a website. The key is that I don't actually want the content, but rather a simple list of URIs. I can get reasonably close to this idea with <a href="http://www.gnu.org/software/wget/" rel="noreferrer">Wget</a> using the <code>--spider</code> option, but when piping that output through a <code>grep</code>, I can't seem to find the right magic to make it work:</p>
  
  <pre><code>wget --spider --force-html -r -l1 http://somesite.com | grep 'Saving to:'
  </code></pre>
  
  <p>The <code>grep</code> filter seems to have absolutely no affect on the <code>wget</code> output. Have I got something wrong or is there another tool I should try that's more geared towards providing this kind of limited result set?</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>So I just found out offline that, by default, <code>wget</code> writes to stderr. I missed that in the man pages (in fact, I still haven't found it if it's in there). Once I piped the return to stdout, I got closer to what I need:</p>
  
  <pre><code>wget --spider --force-html -r -l1 http://somesite.com 2&gt;&amp;1 | grep 'Saving to:'
  </code></pre>
  
  <p>I'd still be interested in other/better means for doing this kind of thing, if any exist.</p>