title: how to find out whether website is using cookies or http based authentication
body: <p>I am trying to automate files download via a webserver. I plan on using wget or curl or python urllib / urllib2. </p>
  
  <p>Most solutions use wget and urllib and urllib2. They all talk of HHTP based authentication and cookie based authentication. My problem is I dont know which one is used in the website that stores my data.
  Here is the interaction with the site:</p>
  
  <ol>
  <li>Normally I login to site <a href="http://www.anysite.com/index.cgi" rel="nofollow">http://www.anysite.com/index.cgi</a>?</li>
  <li>I get a form with a login and password. I type in both and hit return. </li>
  <li>The url stays as <a href="http://www.anysite.com/index.cgi" rel="nofollow">http://www.anysite.com/index.cgi</a>? during the entire interaction. But now I have a list of folders and files</li>
  <li>If I click on a folder or file the URL changes to <a href="http://shamrockstructures.com/cgi-bin/index.cgi?page=download&amp;file=%2Fhome%2Fjanysite%2Fpublic_html%2Fuser_data%2Fuserareas%2Ffile.tar.bz2" rel="nofollow">http://shamrockstructures.com/cgi-bin/index.cgi?page=download&amp;file=%2Fhome%2Fjanysite%2Fpublic_html%2Fuser_data%2Fuserareas%2Ffile.tar.bz2</a></li>
  </ol>
  
  <p>And the browser offers me a chance to save the file</p>
  
  <p>I want to know how to figure out whether the site is using HTTP or cookie based authentication. After which I am assuming I can use cookielib or urllib2 in python to connect to it, get the list of files and folders and recursively download everything while staying connected.</p>
  
  <p>p.S: I have tried the cookie cutter ways to connect via wget and wget --http-user "uname" --http-password "passwd" <a href="http://www.anysite.com/index.cgi" rel="nofollow">http://www.anysite.com/index.cgi</a>? , but they only return the web form back to me.</p>