title: Service Fabric: Consistency when updating state between stateful services
body: <p>Let's imagine there was some code like this in a stateful service:</p>
  
  <pre><code>public async Task&lt;bool&gt; UpdateTheThing()
  {
      using (var tx = StateManager.CreateTransaction())
      {
          await UpdateLocalState(tx);
          // point a
          bool isOK = await otherServiceProxy.UpdateServiceState();
          // point b
          if(isOK)
          {   
              await tx.CommitAsync();
          }
          return isOK;
      }
  }
  </code></pre>
  
  <p>If something bad happens anywhere up to point a we're safe - the transaction will deal with 
  ensuring the data is consistent. If something happens between point a and point b we may
  or may not have updated the state on the other service.</p>
  
  <p>If the bad thing that happened was that an exception was thrown, it's feasable we could
  recover the state on the other service, but if our node was shut down at that point, we'd
  never recover the state.</p>
  
  <p>One way I know to fix a problem like this would be to do the following:</p>
  
  <ol>
  <li>Make UpdateTheThing do nothing other than adding a command to a queue</li>
  <li>Implement some logic to process the queue</li>
  <li>Invent some way to get the result (isOk) back to the caller</li>
  <li>Make UpdateServiceState idempotent</li>
  <li>Make the queue keep retrying the command if there's a problem</li>
  <li>Invent some way to remove the command from the queue if the problem keeps happening and somehow inform someone that there's been a problem.</li>
  </ol>
  
  <p>This is quite a lot of infrastructure to put into place. I'm looking for suggestions of a simpler approach.</p>
  
  <p>I notice that none of the Service Fabric samples seem to deal with this problem.</p>