title: How to programmatically extract information from a web page, using Linux command line?
body: <p>I need to extract the exchange rate of USD to another currency (say, EUR) for a long list of historical dates.</p>
  
  <p>The <code>www.xe.com</code> website gives the historical lookup tool, and using a detailed URL, one can get the rate table for a specific date, w/o populating the <code>Date:</code> and <code>From:</code> boxes. For example, the URL <a href="http://www.xe.com/currencytables/?from=USD&amp;date=2012-10-15" rel="nofollow">http://www.xe.com/currencytables/?from=USD&amp;date=2012-10-15</a> gives the table of conversion rates from USD to other currencies on the day of Oct. 15th, 2012.</p>
  
  <p>Now, assume I have a list of dates, I can loop through the list and change the date part of that URL to get the required page. If I can extract the rates list, then simple <code>grep EUR</code> will give me the relevant exchange rate (I can use awk to specifically extract the rate).</p>
  
  <p>The question is, how can I get the page(s) using Linux command line command? I tried <code>wget</code> but it did not do the job.</p>
  
  <p>If not CLI, is there an easy and straight forward way to programmatically do that (i.e., will require less time than do copy-paste of the dates to the browser's address bar)?</p>
  
  <hr>
  
  <p><strong>UPDATE 1:</strong></p>
  
  <p>When running:</p>
  
  <pre><code>$ wget 'http://www.xe.com/currencytables/?from=USD&amp;date=2012-10-15'
  </code></pre>
  
  <p>I get a file which contain:</p>
  
  <pre><code>&lt;HTML&gt;
  &lt;HEAD&gt;&lt;TITLE&gt;Autoextraction Prohibited&lt;/TITLE&gt;&lt;/HEAD&gt;
  &lt;BODY&gt;
  Automated extraction of our content is prohibited.  See &lt;A HREF="http://www.xe.com/errors/noautoextract.htm"&gt;http://www.xe.com/errors/noautoextract.htm&lt;/A&gt;.
  &lt;/BODY&gt;
  &lt;/HTML&gt;
  </code></pre>
  
  <p>so it seems like the server can identify the type of query and blocks the <code>wget</code>. Any way around this?</p>
  
  <hr>
  
  <p><strong>UPDATE 2:</strong></p>
  
  <p>After reading the response from the <code>wget</code> command and the comments/answers, I checked the ToS of the website and found this clause:</p>
  
  <pre><code>You agree that you shall not:
  ...
  f. use any automatic or manual process to collect, harvest, gather, or extract
     information about other visitors to or users of the Services, or otherwise
     systematically extract data or data fields, including without limitation any
     financial and/or currency data or e-mail addresses;
  </code></pre>
  
  <p>which, I guess, concludes the efforts in this front.</p>
  
  <hr>
  
  <p>Now, for my curiosity, if <code>wget</code> generates an HTTP request, how does the server know that it was a command and not a browser request?</p>