title: Wget Curl - Download entire site and run concordance
body: <p>This might be a strange question but I'm giving it my best shot.</p>
  
  <p>Summary:
  I have a program (written in Bash and Julia) which will outputs the concordance output for a given set of text (in a file) i.e. you pass the file(s) and it'll output the concordance output like (indentation of the bullet#s I'll take care later):</p>
  
  <pre><code>a. word1 (1001 times), occurred in sentence {2,5,7,11,12,51,122, 711,881,884,.. etc}
  aa. word2 (98 times), occurred in sentence {55,77,79,80,81,110, 121,281,284,.. etc}
  ..
  ..
  az. wordN (12001 times), occurred in sentence {&lt;listofNumbers_comma_separated&gt;}
  aa. wordN1 (55 times), occurred in sentence {&lt;listofNumbers_comma_separated&gt;}
  ab. wordN2 (4 times), occurred in sentence {&lt;listofNumbers_comma_separated&gt;}
  ..
  ..
  az. wordNM (995 times), occurred in sentence {&lt;listofNumbers_comma_separated&gt;}
  aba. wordNN (15 times), occurred in sentence {&lt;listofNumbers_comma_separated&gt;}
  abb. wordNO (25 times), occurred in sentence {&lt;listofNumbers_comma_separated&gt;}
  ..
  ..
  abz. wordNP (20 times), occurred in sentence {&lt;listofNumbers_comma_separated&gt;}
  aca. wordNQ (220 times), occurred in sentence {&lt;listofNumbers_comma_separated&gt;}
  acb. wordNQ (220 times), occurred in sentence {&lt;listofNumbers_comma_separated&gt;}
  ..
  ..
  acz. wordNQ (2220 times), occurred in sentence {&lt;listofNumbers_comma_separated&gt;}
  ada. wordNQ (55 times), occurred in sentence {&lt;listofNumbers_comma_separated&gt;}
  ..
  ..
  and so on..
  </code></pre>
  
  <p>Each bullet has a unique word (which follows regex to determine whether check, in and check-in are three different words or whether check-in can be treated as 2 words (check and in). A user can tweak the regex defined in a configuration file to get the desired word for a bullet. This is all working.
  Similarly, a sentence has various regex which determines whether a sentence ends with a '.' or ';' or '\n' etc. Again, concordance piece is working fine.</p>
  
  <p><strong>My questions:</strong></p>
  
  <ol>
  <li>I want to use wget / curl to get all the text(only) out of a site (entire data) without going into an infinite loop (a link reading another link and making an infinite loop). Saw <a href="http://www.labnol.org/software/wget-command-examples/28750/" rel="nofollow">http://www.labnol.org/software/wget-command-examples/28750/</a> and <a href="http://xahlee.info/linux/wget_curl_tutorial.html" rel="nofollow">http://xahlee.info/linux/wget_curl_tutorial.html</a> and <a href="http://linuxreviews.org/quicktips/wget/" rel="nofollow">http://linuxreviews.org/quicktips/wget/</a></li>
  </ol>
  
  <p>I guess something like <strong>wget -m</strong> <strong>-l 7</strong> -t 2 -w 2 <a href="http://www.website.com" rel="nofollow">http://www.website.com</a> will do (tweak -l # if required) but is there any option which can enhance the performance of retrieving the contents faster?</p>
  
  <ol start="2">
  <li>How to find the list of all the websites available in WWW (World Wide Web)? Is it available somewhere where I can run wget in sequence/parallel fashion?</li>
  </ol>
  
  <p>End goal (for fun but actually I'm really curious about it) is to run concordance on every such website file(s) where the site allows you to read data/text in some fashion (combined as my script is parameter driven to run concordance on each file entered or a file containing a list of files which contains text in it) on every site in word wide web. OK, that might scare someone but I think it's not impossible.</p>
  
  <p>So far running my program in reading whole SVN redbook ebook / bible / online novel is working pretty good (2-10 seconds) but now I want to harness it on WWW site contents. I randomly checked some rare words (appendix) and they match with output generated by the concordance script/program. </p>
  
  <p>Has anybody tried this on a minimal scale i.e. running concordance on a company's website only or all site which ends with .in (India) etc.</p>