title: Shell script using curl to loop through urls
body: <p>I've been trying to create a simple script that will take a list of queries from a .txt file, append the main url variable, then scrape the content and output it to a text file.</p>
  
  <p>Here's what I have so far:</p>
  
  <pre><code>#!/bin/bash
  
  url="example.com/?q="
  for i in $(cat query.txt); do
      content=$(curl -o $url $i)
      echo $url $i
      echo $content &gt;&gt; output.txt
  done
  </code></pre>
  
  <p>list:</p>
  
  <pre><code>images
  news
  stuff
  other
  </code></pre>
  
  <p>error log:</p>
  
  <pre><code>curl: (6) Could not resolve host: other; nodename nor servname provided, or not known
  example.com/?q= other
  </code></pre>
  
  <p>If I use this command straight from the command line I get some output into the file:</p>
  
  <pre><code>curl -L http://example.com/?q=other &gt;&gt; output.txt
  </code></pre>
  
  <p>Ultimately I would like the output to be:</p>
  
  <pre><code>fetched:    http://example.com/?q=other
  content:    the output of the page
  
  followed by the next query in the list.
  </code></pre>