Consistent way to handle transient timeouts with WCF calls (timeouts, unreliable network, server load, etc) I'm currently using the following code to manage calls to WCF services that are unreliable, and or suffer performance load issues due to contention.

Right now this code is limited to solving issue with types that return void. It exponentially backs off from a subsequent call from the server, preventing unhelpful "hammering" of the server.

Question 


Is there any possible way I can abstract this into something more generic for varying return types?
Have I caught all the appropriate exceptions worthy of a retry?
What is the most appropriate way I can communicate to the calling process what is going on with this code, perhaps offering an opportunity to cancel early?
Any other enhancements you may think of...


Usage Example:

Service&lt;IOrderService&gt;.Use(orderService=&gt;
{
  orderService.PlaceOrder(request);
}


Source code

public delegate void UseServiceDelegate&lt;T&gt;(T proxy);

public static class Service&lt;T&gt;
{
    public static ChannelFactory&lt;T&gt; _channelFactory = new ChannelFactory&lt;T&gt;("");

    public static void Use(UseServiceDelegate&lt;T&gt; codeBlock)
    {
        IClientChannel proxy = (IClientChannel)_channelFactory.CreateChannel();
        bool success = false;


       Exception mostRecentEx = null;
       int millsecondsToSleep = 1000;

       for(int i=0; i&lt;5; i++)  // Attempt a maximum of 5 times
       {
           try
           {
               codeBlock((T)proxy);
               proxy.Close();
               success = true;
               break;
           }

           // The following is typically thrown on the client when a channel is terminated due to the server closing the connection.
           catch (ChannelTerminatedException cte)
           {
              mostRecentEx = cte;
               proxy.Abort();
               //  delay (backoff) and retry
               Thread.Sleep(millsecondsToSleep  * (i + 1));
           }

           // The following is thrown when a remote endpoint could not be found or reached.  The endpoint may not be found or
           // reachable because the remote endpoint is down, the remote endpoint is unreachable, or because the remote network is unreachable.
           catch (EndpointNotFoundException enfe)
           {
              mostRecentEx = enfe;
               proxy.Abort();
               //  delay (backoff) and retry
               Thread.Sleep(millsecondsToSleep * (i + 1));
           }

           // The following exception that is thrown when a server is too busy to accept a message.
           catch (ServerTooBusyException stbe)
           {
              mostRecentEx = stbe;
               proxy.Abort();

               //  delay (backoff) and retry
               Thread.Sleep(millsecondsToSleep * (i + 1));
           }
           catch (TimeoutException timeoutEx)
           {
               mostRecentEx = timeoutEx;
               proxy.Abort();

               //  delay (backoff) and retry
               Thread.Sleep(millsecondsToSleep * (i + 1));
           }
           catch (CommunicationException comException)
           {
               mostRecentEx = comException;
               proxy.Abort();

               //  delay (backoff) and retry
               Thread.Sleep(millsecondsToSleep * (i + 1));
           }
           catch(Exception )
           {
                // rethrow any other exception not defined here
                // You may want to define a custom Exception class to pass information such as failure count, and failure type
                proxy.Abort();
                throw ; 
           }
       }
       if (success == false &amp;&amp; mostRecentEx != null)
       {
           proxy.Abort();
           throw new Exception("WCF call failed after 5 retries.", mostRecentEx );
       }

    }
}
One clean up is to consolidate the common code in the catch blocks, by putting it into a function. Other ideas:


5 tries is not many for WCF in general. I would increase it to 50 or so. If you do this, you need a ceiling to multiple applied to to the minimum sleep time.
The milliseconds to sleep should be initialized to a semi-random value: a minimum value, plus a random fraction of an additional amount. Usually I make the additional amount equal to the minimum value. This will keep all your clients from hammering the server with the same rhythm, in the event all clients start at the same time.


With regards to questions:


Make the method generic, with a type parameter defining the return value.
I usually have only two catch blocks, one for FaultException to communicate exceptions explicitly from server to client, and one for Exception, to catch everything else. The Exception block is where you initiate a retry. Read about FaultContractAttribute.
The only good way to allow cancellation is to run all of this in a background thread, possibly using BackgroundWorker.