title: Mirror WebSite download specific file type with BASH
body: <p>I am trying to archive collections from several websites. I want to be able to maintain them is some sort of organization. So it would be ideal to store them in a mirrored dir structure. Below is my attempt</p>
  
  <pre><code>wget -m -x -e robots=off --no-parent --accept "*.ext" http://example.com
  </code></pre>
  
  <p>While using the "-m" option does it have any limit on how far it goes? (will it wander off the the web-site? go forever?) if so, is it better to use</p>
  
  <pre><code>wget -r -x -e robots=off --no-parent --accept "*.ext" --level 2 http://example.com
  </code></pre>
  
  <p>Is this the most reasonable way to do this? I know that "wget" has a --spider option, Is it stable?</p>
  
  <blockquote>
    <p>EDIT</p>
  </blockquote>
  
  <p><strong>This is the solution I have found.</strong></p>
  
  <p>The files I was looking for are tagged and stored in a single <code>dir</code> on the server side. When trying variations of <code>wget</code>. I was able to get structure of the links and various files but I kept having trouble with links running in loops. So I came up with this work around. It works but is slow. Any advice on how to increase efficiency?</p>
  
  <p><strong>The structure of website &amp; files that I am trying to get</strong></p>
  
  <pre><code>home
     ├──Foo
     │  ├──paul.mp3
     │  ├──saul.mp3
     │  ├──micheal.mp3
     │  ├──ring.mp3
     ├──Bar
        ├──nancy.mp3
        ├──jan.mp3
        ├──mary.mp3
  </code></pre>
  
  <p>So first I created,
  <strong>The file with the TAGS of the files that I want</strong></p>
  
  <pre><code>taglist.txt
  foo
  bar
  </code></pre>
  
  <p><strong>The Script</strong></p>
  
  <pre><code>#!/bin/bash
  
  #this script seems to work until the download part
  
  
  URL="http://www.example.com"
  LINK_FILE=taglist.txt
  
  while read TAG; do
      mkdir "$TAG"
      cd "$TAG"
  
          # Get the URLs from the page
          wget -q $URL/$TAG -O - | \tr "\t\r\n'" '   "' | \grep -i -o '&lt;a[^&gt;]\+href[ ]*=[ \t]*"\(ht\|f\)tps\?:[^"]\+"' | \sed -e 's/^.*"\([^"]\+\)".*$/\1/g' &gt; tmp.urls.txt
          # Clean and sort URLs
          grep -i 'http://www.example.com/storage_dir/*' tmp.urls.txt | sort -u &gt; tmp.curls.txt
              # Download the page with the URL
              while read TAPE_URL; do
              #wget -r -A.mp3 $TAPE_URL
              wget -O tmp.$RANDOM $TAPE_URL
              done &lt;tmp.curls.txt
              # Find all the .mp3 links in the files
              grep -r -o -E 'href="([^"#]+)[.mp3]"' * | cut -d'"' -f2 | sort | uniq &gt; $TAG.mp3.list
              # Clean Up
              rm tmp.* 
              # Download the collected URLs
              wget -i $TAG.mp3.list
      cd ..   
  done &lt;"$LINK_FILE"
  </code></pre>