title: Exception when downloading data from HTTPS site
body: <p>I am working on a siteripper / screenscraper for looking up tracking information on the Royal Mail website. Unfortunately Royal Mail do not support an API, so this is the way to do it.</p>
  
  <p>I keep getting the same exception no matter what I do.
  (<strong>The remote server returned an error: (500) Internal Server Error.</strong>)</p>
  
  <p>My base code is:</p>
  
  <pre><code>class Program
  {
      static void Main(string[] args)
      {
          string url = "http://track.royalmail.com/portal/rm/track?catId=22700601&amp;gear=authentication&amp;forcesegment=SG-Personal";
          byte[] response;
  
          WebClient webClient = new WebClient();
          response = webClient.DownloadData(url);
      }
  }
  </code></pre>
  
  <p>I have used Fiddler, to investigate the data transactions made by my browser in order to mimic that in my code. I can see Royal Mail uses cookies, so I have tried to implement a WebClient that supports cookies by adding a cookie handler to it:</p>
  
  <pre><code>public class CookieAwareWebClient : WebClient
  {
      private CookieContainer m_container = new CookieContainer();
  
      protected override WebRequest GetWebRequest(Uri address)
      {
          WebRequest request = base.GetWebRequest(address);
          if (request is HttpWebRequest)
          {
              (request as HttpWebRequest).CookieContainer = m_container;
          }
          return request;
      }
  }
  </code></pre>
  
  <p>But that didn't help eather :-(</p>
  
  <p>I have also tried to look up the tracking information through Royal Mails SSL protected site (<a href="https://www.royalmail.com/portal/sme/track?catId=62200738&amp;mediaId=63900708" rel="nofollow">https://www.royalmail.com/portal/sme/track?catId=62200738&amp;mediaId=63900708</a>), and implementing credentials into my C# program, but no luck there.</p>
  
  <p>I have now meet the wall, and I keep bumping into the same tutorials / threads that don't seem to help me any further.</p>
  
  <p>I hope there is a brilliant brain out there :-)</p>