title: powershell httpwebrequest GET method cookiecontainer problem?
body: <p>I'm trying to scrape a website that has user authentication. I am able to do a POST to send my login and stores a cookie. However, after the login I get a 403 error when trying to access the protected page.</p>
  
  <pre><code>$url = "https://some_url"
  
  $CookieContainer = New-Object System.Net.CookieContainer
  
  $postData = "User=UserName&amp;Password=Pass"
  
  $buffer = [text.encoding]::ascii.getbytes($postData)
  
  [net.httpWebRequest] $req = [net.webRequest]::create($url)
  $req.method = "POST"
  $req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
  $req.Headers.Add("Accept-Language: en-US")
  $req.Headers.Add("Accept-Encoding: gzip,deflate")
  $req.Headers.Add("Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7")
  $req.AllowAutoRedirect = $false
  $req.ContentType = "application/x-www-form-urlencoded"
  $req.ContentLength = $buffer.length
  $req.TimeOut = 50000
  $req.KeepAlive = $true
  $req.Headers.Add("Keep-Alive: 300");
  $req.CookieContainer = $CookieContainer
  $reqst = $req.getRequestStream()
  $reqst.write($buffer, 0, $buffer.length)
  $reqst.flush()
  $reqst.close()
  [net.httpWebResponse] $res = $req.getResponse()
  $resst = $res.getResponseStream()
  $sr = new-object IO.StreamReader($resst)
  $result = $sr.ReadToEnd()
  $res.close()
  
  
  
  $url2 = "https://some_url/protected_page"
  
  [net.httpWebRequest] $req2 = [net.webRequest]::create($url2)
  $req2.Method = "GET"
  $req2.Accept = "text/html"
  $req2.AllowAutoRedirect = $false
  $req2.CookieContainer = $CookieContainer
  $req2.TimeOut = 50000
  [net.httpWebResponse] $res2 = $req2.getResponse()
  $resst = $res2.getResponseStream()
  $sr = new-object IO.StreamReader($resst)
  $result = $sr.ReadToEnd()
  </code></pre>
  
  <p><strong>WORKAROUND:</strong> So after trying almost everything I ended up trying something different and it actually works.</p>
  
  <p>After posting the login and getting the session cookie, I use webclient to access the secure page by adding the cookie string to the headers.</p>
  
  <pre><code>$web = new-object net.webclient
  $web.Headers.add("Cookie", $res.Headers["Set-Cookie"])
  $result = $web.DownloadString("https://secure_url")
  </code></pre>
  
  <p>One of the cool thing about this is that webclient saves the cookie. To access another secure page, you can just call $web.downloadstring("https://another_secure_url") :)</p>