title: Super slow user provisioning with errors in the log of Azure AD Connect Provisioning Agent
body: <p>I am having trouble with the Azure AD Connect Provisioning Agent recently. The user provisioning can take 2 hours and the status goes to Quarantine sometimes because of timeout. Eventually the sync succeeds but I would like to find out why it takes so long and the causes of errors.</p>
  <p>In the log of the agent (C:\ProgramData\Microsoft\Azure AD Connect Provisioning Agent\Trace) I see this error over and over with a lot of stack traces:</p>
  <blockquote>
  <p>ADConnectProvisioningAgent.exe Error: 0 : Exception:
  System.Net.WebSockets.WebSocketException (0x80004005): Failed
  establishing web socket connection. ---&gt; System.Net.WebException: The
  remote server returned an error: (400) Bad Request.</p>
  </blockquote>
  <p>What can I do to troubleshoot this? The agent is installed on a Domain Controller Azure VM.</p>
  <p>I turned on Verbose logs with Start-AADCloudSyncToolsVerboseLogs and I caught the time between failing and success (same error message before over and over, and after the sync is starting properly and no more errors), see below the snippet where the transition occurs:</p>
  <p>Thanks!</p>
  <pre><code>AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:12:43.9366855Z](72) AADConnectProvisioningAgent.exe Information: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:12:43.9366855Z](72) Waiting '1600'ms before next retry. 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:12:45.5508737Z](88) AADConnectProvisioningAgent.exe Information: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:12:45.5508737Z](88) Establishing web socket connection. ConnectionId: 'ce509fbb-f3db-4ffb-9cb3-05190b27d2b3', TransactionId: '436c268a-bc5b-4665-81bd-180a579066b7', Timeout: '30000' ms.
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:12:45.5508737Z](88) AADConnectProvisioningAgent.exe Information: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:12:45.5508737Z](88) Establishing web socket connection. ConnectionId: 'ce509fbb-f3db-4ffb-9cb3-05190b27d2b3', TransactionId: '436c268a-bc5b-4665-81bd-180a579066b7', Keep-alive interval: '00:00:10'.
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:12:45.5508737Z](88) AADConnectProvisioningAgent.exe Information: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:12:45.5508737Z](88) CacheDnsLookup.TryResolveDnsAsync: Hostname 'vm1-proxy-hisgeneral-EUS-BL02P-2.connector.his.msappproxy.net' resolved to value '52.188.116.56' with lookupState: 'CacheFirstSuccess', CorrelationId: '436c268a-bc5b-4665-81bd-180a579066b7'
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:12:45.6289926Z](72) AADConnectProvisioningAgent.exe Error: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:12:45.6289926Z](72) Error while establishing web socket connection. ConnectionId: 'ce509fbb-f3db-4ffb-9cb3-05190b27d2b3', TransactionId: '436c268a-bc5b-4665-81bd-180a579066b7'. Exception: 'System.Net.WebException: The remote server returned an error: (400) Bad Request.
     at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
     at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.ApplicationProxy.Connector.Listeners.ConnectorSignalingWebSocket.&lt;ConnectAsync&gt;d__49.MoveNext()'
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:12:45.6289926Z](72) AADConnectProvisioningAgent.exe Error: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:12:45.6289926Z](72) Retryable Operation is rethrowing error after failed with Exception: 'System.Net.WebSockets.WebSocketException (0x80004005): Failed establishing web socket connection. ---&gt; System.Net.WebException: The remote server returned an error: (400) Bad Request.
     at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
     at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.ApplicationProxy.Connector.Listeners.ConnectorSignalingWebSocket.&lt;ConnectAsync&gt;d__49.MoveNext()
     at Microsoft.ApplicationProxy.Connector.Listeners.ConnectorSignalingWebSocket.&lt;ConnectAsync&gt;d__49.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at Microsoft.ApplicationProxy.Common.Utilities.Extensions.TaskExtensions.&lt;TimeoutAfter&gt;d__1.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.ApplicationProxy.Connector.Listeners.ConnectorSignalingWebSocket.&lt;TryEstablishConnectionAsync&gt;d__44.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.ApplicationProxy.Connector.Listeners.ConnectorSignalingWebSocket.&lt;EstablishConnectionAndOpenChannelAsync&gt;d__43.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.ApplicationProxy.Connector.Listeners.ConnectorSignalingWebSocket.&lt;&gt;c__DisplayClass41_0.&lt;&lt;Connect&gt;b__0&gt;d.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.ApplicationProxy.Common.Utilities.Communication.RetryableOperationInvokerBase.&lt;InvokeOperation&gt;d__8`1.MoveNext()'. 
  Attempt '5'. 
  Rethrowing error. 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:12:45.6289926Z](72) AADConnectProvisioningAgent.exe Error: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:12:45.6289926Z](72) Exception: System.Net.WebSockets.WebSocketException (0x80004005): Failed establishing web socket connection. ---&gt; System.Net.WebException: The remote server returned an error: (400) Bad Request.
     at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
     at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.ApplicationProxy.Connector.Listeners.ConnectorSignalingWebSocket.&lt;ConnectAsync&gt;d__49.MoveNext()
     at Microsoft.ApplicationProxy.Connector.Listeners.ConnectorSignalingWebSocket.&lt;ConnectAsync&gt;d__49.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at Microsoft.ApplicationProxy.Common.Utilities.Extensions.TaskExtensions.&lt;TimeoutAfter&gt;d__1.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.ApplicationProxy.Connector.Listeners.ConnectorSignalingWebSocket.&lt;TryEstablishConnectionAsync&gt;d__44.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.ApplicationProxy.Connector.Listeners.ConnectorSignalingWebSocket.&lt;EstablishConnectionAndOpenChannelAsync&gt;d__43.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.ApplicationProxy.Connector.Listeners.ConnectorSignalingWebSocket.&lt;&gt;c__DisplayClass41_0.&lt;&lt;Connect&gt;b__0&gt;d.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.ApplicationProxy.Common.Utilities.Communication.RetryableOperationInvokerBase.&lt;InvokeOperation&gt;d__8`1.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.ApplicationProxy.Connector.Utils.ConnectorWebSocketRetryableOperationInvoker.&lt;InvokeWithRetry&gt;d__0`1.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.ApplicationProxy.Connector.Listeners.ConnectorSignalingWebSocket.&lt;Connect&gt;d__41.MoveNext()
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:12:45.6289926Z](72) AADConnectProvisioningAgent.exe Information: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:12:45.6289926Z](72) Connection with server closed. ConnectionId, 'ce509fbb-f3db-4ffb-9cb3-05190b27d2b3'
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:12:45.6289926Z](73) AADConnectProvisioningAgent.exe Information: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:12:45.6289926Z](73) ConnectorWebSocket finished. TransactionId: '436c268a-bc5b-4665-81bd-180a579066b7', ConnectionId: 'ce509fbb-f3db-4ffb-9cb3-05190b27d2b3'
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.2544225Z](37) AADConnectProvisioningAgent.exe Information: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.2544225Z](37) Client request received via Relay, TransactionId: '46907b7c-4084-49db-9381-2a6bcdfb8b69', Configuration Hash : '-1377476654'
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.2544225Z](37) AADConnectProvisioningAgent.exe Information: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.2544225Z](37) Returning Ack. Ack duration: '0', TransactionId: '46907b7c-4084-49db-9381-2a6bcdfb8b69'
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.2544225Z](52) AADConnectProvisioningAgent.exe Information: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.2544225Z](52) Connector Request for TransactionId '46907b7c-4084-49db-9381-2a6bcdfb8b69' and SessionId '00000000-0000-0000-0000-000000000000' was created. Primary Host and port are 'vm2-proxy-hisgeneral-EUS-BL02P-2.connector.his.msappproxy.net':'443'. Fallback host and port are 'vm2-proxy-hisgeneral-EUS-BL02P-2.connector.his.msappproxy.net':'10100'
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.2544225Z](52) AADConnectProvisioningAgent.exe Information: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.2544225Z](52) Establishing web socket connection. ConnectionId: '9a5a44d6-c202-4e96-a004-5648df920b6e', TransactionId: '46907b7c-4084-49db-9381-2a6bcdfb8b69', Timeout: '30000' ms.
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.2544225Z](52) AADConnectProvisioningAgent.exe Information: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.2544225Z](52) Establishing web socket connection. ConnectionId: '9a5a44d6-c202-4e96-a004-5648df920b6e', TransactionId: '46907b7c-4084-49db-9381-2a6bcdfb8b69', Keep-alive interval: '00:00:10'.
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.2544225Z](52) AADConnectProvisioningAgent.exe Information: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.2544225Z](52) CacheDnsLookup.TryResolveDnsAsync: Hostname 'vm2-proxy-hisgeneral-EUS-BL02P-2.connector.his.msappproxy.net' resolved to value '40.117.125.47' with lookupState: 'CacheFirstSuccess', CorrelationId: '46907b7c-4084-49db-9381-2a6bcdfb8b69'
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.4107247Z](37) AADConnectProvisioningAgent.exe Information: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.4107247Z](37) Starting web socket listener. ConnectionId, '9a5a44d6-c202-4e96-a004-5648df920b6e'
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.4107247Z](37) AADConnectProvisioningAgent.exe Information: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.4107247Z](37) Sending web socket 'Connect' message. ConnectionId, '9a5a44d6-c202-4e96-a004-5648df920b6e'
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.4575414Z](41) AADConnectProvisioningAgent.exe Information: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.4575414Z](41) Received web socket signaling message of type 'TrustedParty' protocol.
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.4575414Z](40) AADConnectProvisioningAgent.exe Information: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.4575414Z](40) Connector Request for TransactionId '46907b7c-4084-49db-9381-2a6bcdfb8b69' and SessionId '00000000-0000-0000-0000-000000000000' was created. Primary Host and port are 'vm2-proxy-hisgeneral-EUS-BL02P-2.connector.his.msappproxy.net':'443'. Fallback host and port are 'vm2-proxy-hisgeneral-EUS-BL02P-2.connector.his.msappproxy.net':'10100'
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.4731616Z](40) AADConnectProvisioningAgent.exe Information: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.4731616Z](40) TrustedPartyRequestProcessor: Processing request '46907b7c-4084-49db-9381-2a6bcdfb8b69' in session '00000000-0000-0000-0000-000000000000'
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.4731616Z](40) AADConnectProvisioningAgent.exe Information: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.4731616Z](40) TrustedPartyRequestProcessor: TrustedPartyName 'syncfabric', ScenarioName 'aadprovisioning', ScenarioVersion 'v1', RequestBody ''
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.4887888Z](40) AADConnectProvisioningAgent.exe Information: 0 : 
  AAD Connect Provisioning Agent Information: 5 : [2021-10-01T14:18:29.4887888Z](40) CacheDnsLookup.TryResolveDnsAsync: Hostname 'vm2-proxy-hisgeneral-eus-bl02p-2.connector.his.msappproxy.net' resolved to value '40.117.125.47' with lookupState: 'CacheFirstSuccess', CorrelationId: '46907b7c-4084-49db-9381-2a6bcdfb8b69'
  AAD Connect Provisioning Agent Information: 2 : [2021-10-01T14:18:29.6294197Z](41) Received message:
  Activity Identifier: 259d1767-723d-4f7e-af1f-a99161832f21
  POST /scim/.search HTTP/1.1
  Host: p
  Content-Type: application/scim+json; charset=utf-8
  </code></pre>