title: WCF - Random client timeout when making multiple calls
body: <p>I have a <code>WPF</code> client requesting data via <code>WCF</code> service hosted in <code>IIS 7</code>. The service method makes a call to a stored procedure (<code>SQL 2012</code>) using <code>EF</code> to retrieve some data.</p>
  
  <p>There's a lot of data to load so the client makes several calls to the service method in an effort to "break up" the data loading and avoid large payloads and timeouts.</p>
  
  <p>We use the generated service proxies that extend from <code>System.ServiceModel.ClientBase&lt;T&gt;.</code></p>
  
  <p>We're also using a custom http binding with binary encoding (from <a href="http://architects.dzone.com/articles/wcf-over-https-compression-and" rel="nofollow noreferrer">here</a>) - actual implementation shown here:</p>
  
  <pre><code>&lt;customBinding&gt;
     &lt;binding name="CustomBinding_IPointDataAccess" closeTimeout="00:01:00"
        openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"&gt;
        &lt;binaryMessageEncoding maxReadPoolSize="64" maxWritePoolSize="16" maxSessionSize="2048"&gt;
        &lt;readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="16384" /&gt;
        &lt;/binaryMessageEncoding&gt;
        &lt;httpTransport manualAddressing="false" maxBufferPoolSize="524288"
           maxReceivedMessageSize="2147483647" allowCookies="false" authenticationScheme="Anonymous" bypassProxyOnLocal="false" decompressionEnabled="true" hostNameComparisonMode="StrongWildcard" keepAliveEnabled="true" maxBufferSize="2147483647" proxyAuthenticationScheme="Anonymous" realm="" transferMode="Buffered" unsafeConnectionNtlmAuthentication="false" useDefaultWebProxy="true" /&gt;
     &lt;/binding&gt;
  </code></pre>
  
  <p></p>
  
  <p>Also, dynamic compression is turned on in IIS. I can view the requests in Fiddler, and the size of the message's body is fine (~50KB) and <em>99% of the requests return in a second or two</em>. Perfect! </p>
  
  <p>However, with almost every iteration, there is one call in the bunch that takes minutes to complete, and I don't know why... My <code>sendTimeOut</code> on the client was at 1 minute and naturally that one call would fail. I extended it to 10 minutes, and the call seems to complete in a little more than 2 minutes - although sometimes it would take even longer. The problem appears very random - it could be the first call, it could be the 30th call. But it's very reproducible.</p>
  
  <p>I placed some logging around the the stored procedure call in the WCF service method and it executes and gets data back in under a second. So, I don't think it's a database issue.</p>
  
  <p>Using Fiddler, the problematic call generates output similar to the following:</p>
  
  <pre><code>ACTUAL PERFORMANCE
  --------------
  ClientConnected:     14:02:42.959
  ClientBeginRequest:  14:03:01.224
  GotRequestHeaders:   14:03:01.224
  ClientDoneRequest:   14:03:01.574
  Determine Gateway:   0ms
  DNS Lookup:      0ms
  TCP/IP Connect:  46ms
  HTTPS Handshake:     0ms
  ServerConnected:     14:05:16.021
  FiddlerBeginRequest: 14:05:16.021
  ServerGotRequest:    14:05:16.021
  ServerBeginResponse: 14:03:04.784
  GotResponseHeaders:  14:05:16.561
  ServerDoneResponse:  14:05:16.611
  ClientBeginResponse: 14:05:16.611
  ClientDoneResponse:  14:05:16.611
  </code></pre>
  
  <p>Notice the significant time between <code>ServerBeginResponse</code> and <code>GotResponseHeaders</code>. This seems strikingly similar to the issue seen <a href="https://stackoverflow.com/questions/14872999/understanding-fiddler-statistics">here</a>.</p>
  
  <p>I enabled WCF Service Tracing, and at a quick glance, there are no errors or warnings, but I can't really make much sense out of what I'm looking at beyond the basics.</p>
  
  <p><strong>How can I nail down what and where the problem lies? Is it serialization? Is it a network issue? Can the server not keep up with the client sending so many requests?</strong></p>
  
  <p>I've tried adjusting the WCF Throttling in the config file by adding the appropriate <code>serviceBehaviors</code>, but this did not make a difference.</p>
  
  <p>I should mention that I am doing this over a VPN connection, but other things like file transfers, remote desktop connections work just fine. It seems pretty reliable.</p>
  
  <p>I can provide more details if necessary.</p>
  
  <p><strong>Edit (6.10.2013):</strong> Not sure if this is related or just a fluke, but a couple times, I've noticed that on the problematic call, the Body size is significantly less than the others. This is not the case each time, but it may provide some clues. Here is a screen capture from Fiddler to show you how consistent the Body size should be with each call. The selected entry (#21) is much smaller than the others in size, but takes over 2 minutes to complete. </p>
  
  <p><img src="https://i.stack.imgur.com/DOX66.png" alt="enter image description here"></p>
  
  <p>Oddly enough, this time I received an exception. The exception does not happen every time.</p>
  
  <pre><code>System.ServiceModel.CommunicationException: The server did not provide a meaningful reply; this might be caused by a contract mismatch, a premature session shutdown or an internal server error.
  
  Server stack trace: 
     at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
     at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs)
     at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
     at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
  
  Exception rethrown at [0]: 
     at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
     at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&amp; msgData, Int32 type)
  </code></pre>