title: New-WebServiceProxy failing to authenticate with NTLM
body: <p>I'm dealing with a rather peculiar issue. We have a need to hit the Lists service on our SharePoint farm. Web authentication federated through an Oracle SSO, but we do have accounts configured for automation that can perform web requests. Using AAM, we have an "internal" URL configured for server side automation that bypasses directly to AD, and everything else gets pushed to the SSO.</p>
  
  <p>Here's the code (sanitized) that I'm using to try to get the list collection.</p>
  
  <pre><code>$username = "DOMAIN\username"
  $password = "somepassword"
  $site = "https://sp.biz.com/sites/SiteCollection"
  
  $credentials = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $username, (ConvertTo-SecureString $password -AsPlainText -Force)
  
  $proxy = New-WebServiceProxy -Uri "$site/_vti_bin/Lists.asmx" -Credentials $credentials
  
  $proxy.GetListCollection()
  </code></pre>
  
  <p>I'm hit with a 403 when I use that code.</p>
  
  <blockquote>
    <p>Exception calling "GetListCollection" with "0" argument(s): "Server was unable to process request. ---> Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))"</p>
  </blockquote>
  
  <p>If I change $site to use the internal URL (set via AAM) and run that on one of the front ends, I receive the list collection successfully. Now, at first I thought there was an issue with the account and permissions, but after running a Fiddler capture I see it not authenticating at all.</p>
  
  <p>When I run the following cURL command, it authenticates and returns the list collection. Soap.xml is just the basic GetListCollection packet copied straight from the WDSL.</p>
  
  <pre><code>curl -v -u 'username':'pass' --ntlm -X POST -H "Content-Type: text/xml" --data-binary @soap.xml https://sp.biz.com/sites/SiteCollection/_vti_bin/Lists.asmx
  </code></pre>
  
  <p>Here's the sanitized verbose output from cURL.</p>
  
  <pre><code>* STATE: INIT =&gt; CONNECT handle 0x600056190; line 1029 (connection #-5000)
  * Hostname was NOT found in DNS cache
  *   Trying &lt;IPv6&gt;...
  * STATE: CONNECT =&gt; WAITCONNECT handle 0x600056190; line 1082 (connection #0)
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to sp.biz.com (&lt;IPv6&gt;) port 443 (#0)
  * successfully set certificate verify locations:
  *   CAfile: /usr/ssl/certs/ca-bundle.crt
    CApath: none
  * SSLv3, TLS handshake, Client hello (1):
  } [data not shown]
  * STATE: WAITCONNECT =&gt; PROTOCONNECT handle 0x600056190; line 1222 (connection #0)
  * SSLv3, TLS handshake, Server hello (2):
  { [data not shown]
  * SSLv3, TLS handshake, CERT (11):
  { [data not shown]
  * SSLv3, TLS handshake, Server finished (14):
  { [data not shown]
  * SSLv3, TLS handshake, Client key exchange (16):
  } [data not shown]
  * SSLv3, TLS change cipher, Client hello (1):
  } [data not shown]
  * SSLv3, TLS handshake, Finished (20):
  } [data not shown]
  * SSLv3, TLS change cipher, Client hello (1):
  { [data not shown]
  * SSLv3, TLS handshake, Finished (20):
  { [data not shown]
  * SSL connection using TLSv1.2 / DES-CBC3-SHA
  *        SSL certificate verify ok.
  * STATE: PROTOCONNECT =&gt; DO handle 0x600056190; line 1241 (connection #0)
  * Server auth using NTLM with user 'DOMAIN\username'
  &gt; POST /sites/SiteCollection/_vti_bin/Lists.asmx HTTP/1.1
  &gt; Authorization: NTLM &lt;snip&gt;
  &gt; User-Agent: curl/7.39.0
  &gt; Host: sp.biz.com
  &gt; Accept: */*
  &gt; Content-Type: text/xml
  &gt; Content-Length: 0
  &gt;
  * STATE: DO =&gt; DO_DONE handle 0x600056190; line 1314 (connection #0)
  * STATE: DO_DONE =&gt; WAITPERFORM handle 0x600056190; line 1441 (connection #0)
  * STATE: WAITPERFORM =&gt; PERFORM handle 0x600056190; line 1454 (connection #0)
  * HTTP 1.1 or later with persistent connection, pipelining supported
  &lt; HTTP/1.1 401 Unauthorized
  * Server Microsoft-IIS/7.5 is not blacklisted
  &lt; Server: Microsoft-IIS/7.5
  &lt; SPRequestGuid: &lt;snip&gt;
  &lt; WWW-Authenticate: NTLM &lt;snip&gt;
  &lt; X-Powered-By: ASP.NET
  &lt; MicrosoftSharePointTeamServices: 14.0.0.7006
  &lt; X-MS-InvokeApp: 1; RequireReadOnly
  &lt; Date: Fri, 16 Jan 2015 01:02:56 GMT
  &lt; Content-Length: 0
  &lt; Set-Cookie: BIGipServerserver_pool=&lt;snip&gt;; expires=Sat, 17-Jan-2015 01:02:56 GMT; path=/
  &lt;
    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  * Connection #0 to host sp.biz.com left intact
  * Issue another request to this URL: 'https://sp.biz.com/sites/SiteCollection/_vti_bin/Lists.asmx'
  * STATE: PERFORM =&gt; CONNECT handle 0x600056190; line 1601 (connection #-5000)
  * Found bundle for host sp.biz.com: 0x60006aef0
  * Re-using existing connection! (#0) with host sp.biz.com
  * Connected to sp.biz.com (&lt;IPv6&gt;) port 443 (#0)
  * STATE: CONNECT =&gt; DO handle 0x600056190; line 1075 (connection #0)
  * Server auth using NTLM with user 'DOMAIN\username'
  &gt; POST /sites/SiteCollection/_vti_bin/Lists.asmx HTTP/1.1
  &gt; Authorization: NTLM &lt;snip&gt;
  &gt; User-Agent: curl/7.39.0
  &gt; Host: sp.biz.com
  &gt; Accept: */*
  &gt; Content-Type: text/xml
  &gt; Content-Length: 353
  &gt;
  } [data not shown]
  * upload completely sent off: 353 out of 353 bytes
  * STATE: DO =&gt; DO_DONE handle 0x600056190; line 1314 (connection #0)
  * STATE: DO_DONE =&gt; WAITPERFORM handle 0x600056190; line 1441 (connection #0)
  * STATE: WAITPERFORM =&gt; PERFORM handle 0x600056190; line 1454 (connection #0)
  * HTTP 1.1 or later with persistent connection, pipelining supported
  &lt; HTTP/1.1 200 OK
  &lt; Cache-Control: private, max-age=0
  &lt; Content-Type: text/xml; charset=utf-8
  * Server Microsoft-IIS/7.5 is not blacklisted
  &lt; Server: Microsoft-IIS/7.5
  &lt; SPRequestGuid: &lt;snip&gt;
  &lt; Set-Cookie: FedAuth=&lt;snip&gt;; expires=Fri, 16-Jan-2015 08:36:07 GMT; path=/; secure; HttpOnly
  &lt; X-SharePointHealthScore: 0
  &lt; X-AspNet-Version: 2.0.50727
  &lt; Persistent-Auth: true
  &lt; X-Powered-By: ASP.NET
  &lt; MicrosoftSharePointTeamServices: 14.0.0.7006
  &lt; X-MS-InvokeApp: 1; RequireReadOnly
  &lt; Date: Fri, 16 Jan 2015 01:02:56 GMT
  &lt; Content-Length: 104088
  &lt; Vary: Accept-Encoding
  &lt;
  { [data not shown]
  * STATE: PERFORM =&gt; DONE handle 0x600056190; line 1626 (connection #0)
  100  101k  100  101k  100   353   219k    762 --:--:-- --:--:-- --:--:--  219k
  * Connection #0 to host sp.biz.com left intact
  </code></pre>
  
  <p>Any assistance is greatly appreciated. I'm not opposed to a C# solution over PowerShell if the cmdlets are lacking.</p>
  
  <hr>
  
  <p><strong>01-16-2015 12:13PM EST Update</strong> - I updated the question to reflect HighlyUnavailable's suggestion and included headers from the Fiddler capture.</p>
  
  <p>Here are the sanitized headers from the PowerShell script:</p>
  
  <pre><code>CONNECT sp.biz.com:443 HTTP/1.1
  Host: sp.biz.com
  Connection: Keep-Alive
  HTTP/1.1 200 Connection Established
  FiddlerGateway: Direct
  StartTime: 12:14:46.372
  Connection: close
  ------------------------------------------------------------------
  GET https://sp.biz.com/sites/SiteCollection/_vti_bin/Lists.asmx HTTP/1.1
  User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 2.0.50727.5485)
  Host: sp.biz.com
  Connection: Keep-Alive
  HTTP/1.1 200 OK
  Cache-Control: private, max-age=0
  Content-Type: text/html; charset=utf-8
  Server: Microsoft-IIS/7.5
  SPRequestGuid: &lt;snip&gt;
  X-SharePointHealthScore: 0
  X-AspNet-Version: 2.0.50727
  X-Powered-By: ASP.NET
  MicrosoftSharePointTeamServices: 14.0.0.7006
  X-MS-InvokeApp: 1; RequireReadOnly
  Date: Fri, 16 Jan 2015 17:14:46 GMT
  Connection: keep-alive
  Content-Length: 9066
  Set-Cookie: BIGipServerserver_pool=&lt;snip&gt;; expires=Sat, 17-Jan-2015 17:14:46 GMT; path=/
  Vary: Accept-Encoding
  ------------------------------------------------------------------
  GET https://sp.biz.com/_vti_bin/Lists.asmx?disco HTTP/1.1
  User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 2.0.50727.5485)
  Host: sp.biz.com
  HTTP/1.1 200 OK
  Cache-Control: private
  Content-Type: text/xml; charset=utf-8
  Server: Microsoft-IIS/7.5
  SPRequestGuid: &lt;snip&gt;
  X-SharePointHealthScore: 0
  X-AspNet-Version: 2.0.50727
  X-Powered-By: ASP.NET
  MicrosoftSharePointTeamServices: 14.0.0.7006
  X-MS-InvokeApp: 1; RequireReadOnly
  Date: Fri, 16 Jan 2015 17:14:46 GMT
  Connection: close
  Content-Length: 747
  ------------------------------------------------------------------
  CONNECT sp.biz.com:443 HTTP/1.1
  Host: sp.biz.com
  Connection: Keep-Alive
  HTTP/1.1 200 Connection Established
  FiddlerGateway: Direct
  StartTime: 12:14:47.505
  Connection: close
  ------------------------------------------------------------------
  GET https://sp.biz.com/_vti_bin/Lists.asmx?wsdl HTTP/1.1
  User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 2.0.50727.5485)
  Host: sp.biz.com
  HTTP/1.1 200 OK
  Cache-Control: private
  Content-Type: text/xml; charset=utf-8
  Server: Microsoft-IIS/7.5
  SPRequestGuid: &lt;snip&gt;
  X-SharePointHealthScore: 0
  X-AspNet-Version: 2.0.50727
  X-Powered-By: ASP.NET
  MicrosoftSharePointTeamServices: 14.0.0.7006
  X-MS-InvokeApp: 1; RequireReadOnly
  Date: Fri, 16 Jan 2015 17:14:46 GMT
  Connection: close
  Content-Length: 72672
  Set-Cookie: BIGipServerserver_pool=&lt;snip&gt;; expires=Sat, 17-Jan-2015 17:14:47 GMT; path=/
  Vary: Accept-Encoding
  ------------------------------------------------------------------
  CONNECT sp.biz.com:443 HTTP/1.1
  Host: sp.biz.com
  Connection: Keep-Alive
  HTTP/1.1 200 Connection Established
  FiddlerGateway: Direct
  StartTime: 12:14:48.727
  Connection: close
  ------------------------------------------------------------------
  POST https://sp.biz.com/_vti_bin/Lists.asmx HTTP/1.1
  User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 2.0.50727.5485)
  Content-Type: text/xml; charset=utf-8
  SOAPAction: "http://schemas.microsoft.com/sharepoint/soap/GetListCollection"
  Host: sp.biz.com
  Content-Length: 321
  Expect: 100-continue
  HTTP/1.1 500 Internal Server Error
  Cache-Control: private
  Content-Type: text/xml; charset=utf-8
  Server: Microsoft-IIS/7.5
  X-AspNet-Version: 2.0.50727
  X-Powered-By: ASP.NET
  MicrosoftSharePointTeamServices: 14.0.0.7006
  X-MS-InvokeApp: 1; RequireReadOnly
  Date: Fri, 16 Jan 2015 17:14:48 GMT
  Content-Length: 459
  Set-Cookie: BIGipServerserver_pool=686493706.47873.0000; expires=Sat, 17-Jan-2015 17:14:48 GMT; path=/
  ------------------------------------------------------------------
  </code></pre>
  
  <p>Here are the headers for the cURL command. </p>
  
  <pre><code>CONNECT sp.biz.com:443 HTTP/1.1
  Host: sp.biz.com:443
  User-Agent: curl/7.39.0
  Connection: Keep-Alive
  Content-Type: text/xml
  HTTP/1.1 200 Connection Established
  FiddlerGateway: Direct
  StartTime: 12:21:07.928
  Connection: close
  ------------------------------------------------------------------
  POST https://sp.biz.com/sites/SiteCollection/_vti_bin/Lists.asmx HTTP/1.1
  Authorization: NTLM &lt;snip&gt;=
  User-Agent: curl/7.39.0
  Host: sp.biz.com
  Accept: */*
  Content-Type: text/xml
  Content-Length: 0
  HTTP/1.1 401 Unauthorized
  Server: Microsoft-IIS/7.5
  SPRequestGuid: &lt;snip&gt;
  WWW-Authenticate: NTLM &lt;snip&gt;
  X-Powered-By: ASP.NET
  MicrosoftSharePointTeamServices: 14.0.0.7006
  X-MS-InvokeApp: 1; RequireReadOnly
  Date: Fri, 16 Jan 2015 17:21:07 GMT
  Content-Length: 0
  Set-Cookie: BIGipServerserver_pool=&lt;snip&gt;; expires=Sat, 17-Jan-2015 17:21:07 GMT; path=/
  Proxy-Support: Session-Based-Authentication
  ------------------------------------------------------------------
  POST https://sp.biz.com/sites/SiteCollection/_vti_bin/Lists.asmx HTTP/1.1
  Authorization: NTLM &lt;snip&gt;
  User-Agent: curl/7.39.0
  Host: sp.biz.com
  Accept: */*
  Content-Type: text/xml
  Content-Length: 417
  HTTP/1.1 200 OK
  Cache-Control: private, max-age=0
  Content-Type: text/xml; charset=utf-8
  Server: Microsoft-IIS/7.5
  SPRequestGuid: &lt;snip&gt;
  Set-Cookie: FedAuth=&lt;snip&gt;; expires=Sat, 17-Jan-2015 03:20:50 GMT; path=/; secure; HttpOnly
  X-SharePointHealthScore: 0
  X-AspNet-Version: 2.0.50727
  Persistent-Auth: true
  X-Powered-By: ASP.NET
  MicrosoftSharePointTeamServices: 14.0.0.7006
  X-MS-InvokeApp: 1; RequireReadOnly
  Date: Fri, 16 Jan 2015 17:21:07 GMT
  Content-Length: 66628
  Vary: Accept-Encoding
  ------------------------------------------------------------------
  </code></pre>