title: wget spider returns all URLs twice -- where is the bug?
body: <p>I was looking for a script to create a URL list for a sitemap and found this one:</p>
  
  <pre><code>wget --spider --force-html -r -l1 http://sld.tld 2&gt;&amp;1 \
    | grep '^--' | awk '{ print $3 }' \
    | grep -v '\.\(css\|js\|png\|gif\|jpg\|ico\|txt\)$' \
    &gt; urllist.txt
  </code></pre>
  
  <p>The result is:</p>
  
  <pre><code>http://sld.tld/
  http://sld.tld/
  http://sld.tld/home.html
  http://sld.tld/home.html
  http://sld.tld/news.html
  http://sld.tld/news.html
  ...
  </code></pre>
  
  <p>Every URL entry is saved twice. How should the script be changed to fix this?</p>