title: Strange exception after IChangeFeedObserver ProcessChangesAsync
body: <p>I'm trying to create replication from CosmosDB to local CouchBase Lite database. For this I'm using Microsoft.Azure.Documents.ChangeFeedProcessor</p>
  
  <p>But for some reason always after IChangeFeedObserver ProcessChangesAsync has ended I get following exception on console:</p>
  
  <p>What does "Errors":["One of the specified pre-condition is not met"] mean? Replication is ended succesfully otherwise but this bothers me. </p>
  
  <p><strong>Updated</strong> with system.diagnostics</p>
  
  <pre><code>ChangeFeedEventHost Information: 53 : 09/25/2017 08:27:54: Host 'TESTIkiosk' starting to check for available leases.
  ChangeFeedEventHost Information: 54 : 09/25/2017 08:27:54: Host 'TESTIkiosk' starting to check for available leases.
  ChangeFeedEventHost Information: 55 : 09/25/2017 08:27:54: Host 'TESTIkiosk' 1 partitions, 1 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 1, will try to take 0 lease(s) for myself'.
  ChangeFeedEventHost Information: 56 : 09/25/2017 08:27:54: Host 'TESTIkiosk' 1 partitions, 1 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 1, will try to take 0 lease(s) for myself'.
  ChangeFeedEventHost Information: 57 : 09/25/2017 08:28:06: Host 'TESTIkiosk' starting renewal of Leases.
  ChangeFeedEventHost Information: 58 : 09/25/2017 08:28:06: Host 'TESTIkiosk' renewing lease for PartitionId '0' with lease token '"0000fa04-0000-0000-0000-59c893d40000"'
  ChangeFeedEventHost Information: 59 : 09/25/2017 08:28:06: Host 'TESTIkiosk' starting renewal of Leases.
  ChangeFeedEventHost Information: 60 : 09/25/2017 08:28:06: Host 'TESTIkiosk' renewing lease for PartitionId '0' with lease token '"0000fb04-0000-0000-0000-59c893d40000"'
  ChangeFeedEventHost Information: 61 : 09/25/2017 08:28:06: Host 'TESTIkiosk' attempted to renew lease for PartitionId '0' and lease token '"0000fa04-0000-0000-0000-59c893d40000"' with result: 'True'
  ChangeFeedEventHost Information: 62 : 09/25/2017 08:28:06: Host 'TESTIkiosk' attempted to renew lease for PartitionId '0' and lease token '"0000fb04-0000-0000-0000-59c893d40000"' with result: 'True'
  ChangeFeedEventHost Information: 63 : 09/25/2017 08:28:07: Host 'TESTIkiosk' starting to check for available leases.
  ChangeFeedEventHost Information: 64 : 09/25/2017 08:28:07: Host 'TESTIkiosk' starting to check for available leases.
  ChangeFeedEventHost Information: 65 : 09/25/2017 08:28:07: Host 'TESTIkiosk' 1 partitions, 1 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 1, will try to take 0 lease(s) for myself'.
  ChangeFeedEventHost Information: 66 : 09/25/2017 08:28:07: Host 'TESTIkiosk' 1 partitions, 1 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 1, will try to take 0 lease(s) for myself'.
  [08:28:15 INF] Change feed in Ingredients: total 1 doc(s)
  [08:28:15 INF] Replicated 2614
  ChangeFeedEventHost Information: 67 : 09/25/2017 08:28:15: Partition '0' update failed because the lease with token '"0000e804-0000-0000-0000-59c893b20000"' was updated by same/this host with token '"00000005-0000-0000-0000-59c893e50000"'. Will retry, 4 retry(s) left.
  ChangeFeedEventHost Information: 68 : 09/25/2017 08:28:15: Checkpoint: partition 0, new continuation '"81"'
  
  
  DocDBTrace Error: 0 : DocumentClientException with status code PreconditionFailed, message: Message: {"Errors":["One of the specified pre-condition is not met"]}, inner exception: null, and response headers: {
  "x-ms-last-state-change-utc": "Sun, 24 Sep 2017 21:03:48.392 GMT",
  "lsn": "954",
  "x-ms-schemaversion": "1.3",
  "x-ms-quorum-acked-lsn": "954",
  "x-ms-current-write-quorum": "3",
  "x-ms-current-replica-set-size": "4",
  "x-ms-documentdb-partitionkeyrangeid": "0",
  "x-ms-xp-role": "1",
  "x-ms-request-charge": "1.24",
  "x-ms-serviceversion": " version=1.17.52.1",
  "x-ms-activity-id": "b9877abb-6203-4408-b1c3-92c9f52aad67",
  }
  DocDBTrace Error: 0 : Operation will NOT be retried. Current attempt 0, Exception: Microsoft.Azure.Documents.PreconditionFailedException: Message: {"Errors":["One of the specified pre-condition is not met"]}
  ActivityId: b9877abb-6203-4408-b1c3-92c9f52aad67, Request URI: /apps/c9c8f510-0ca7-4702-aa6c-9c596d797367/services/75cca430-a307-47a0-89aa-82aabf32e065/partitions/53fb63ca-95ae-4d13-90be-d32f1a88843c/replicas/131504697049132845p/
     at Microsoft.Azure.Documents.TransportClient.ThrowIfFailed(String resourceAddress, StoreResponse storeResponse, Uri physicalAddress, Guid activityId)
     at Microsoft.Azure.Documents.RntbdTransportClient.&lt;InvokeStoreAsync&gt;d__0.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.Azure.Documents.ReplicatedResourceClient.&lt;WriteAsync&gt;d__1b.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.Azure.Documents.ReplicatedResourceClient.&lt;InvokeAsync&gt;d__b.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.Azure.Documents.ReplicatedResourceClient.&lt;&gt;c__DisplayClass1.&lt;&lt;InvokeAsync&gt;b__0&gt;d__3.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.Azure.Documents.BackoffRetryUtility`1.&lt;&gt;c__DisplayClassf`1.&lt;&lt;ExecuteAsync&gt;b__d&gt;d__11.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.Azure.Documents.BackoffRetryUtility`1.&lt;ExecuteRetry&gt;d__1b.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at Microsoft.Azure.Documents.BackoffRetryUtility`1.&lt;ExecuteRetry&gt;d__1b.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.Azure.Documents.BackoffRetryUtility`1.&lt;ExecuteAsync&gt;d__18`1.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.Azure.Documents.ReplicatedResourceClient.&lt;InvokeAsync&gt;d__7.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.Azure.Documents.StoreClient.&lt;ProcessMessageAsync&gt;d__0.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.Azure.Documents.Client.DocumentClient.&lt;UpdateAsync&gt;d__305.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.Azure.Documents.Client.DocumentClient.&lt;ReplaceDocumentPrivateAsync&gt;d__13d.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.Azure.Documents.Client.DocumentClient.&lt;ReplaceDocumentPrivateAsync&gt;d__135.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.Azure.Documents.BackoffRetryUtility`1.&lt;&gt;c__DisplayClass2.&lt;&lt;ExecuteAsync&gt;b__0&gt;d__4.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at Microsoft.Azure.Documents.BackoffRetryUtility`1.&lt;ExecuteRetry&gt;d__1b.MoveNext() 
  </code></pre>
  
  <p>Code for my Implementation:</p>
  
  <pre><code>    using Microsoft.Azure.Documents.ChangeFeedProcessor;
      using System;
      using System.Collections.Generic;
      using System.Threading;
      using System.Threading.Tasks;
  
      namespace Solteq.Pos.Repositories.Feeds
      {
          internal class MyHostFeed : IChangeFeedObserver
          {
              private int s_totalDocs = 0;
              private Func&lt;dynamic, bool&gt; replicate;
              private string name;
  
              public Func&lt;dynamic, bool&gt; Replicate { get =&gt; replicate; set =&gt; replicate = value; }
              public string Name { get =&gt; name; set =&gt; name = value; }
  
              public Task OpenAsync(ChangeFeedObserverContext context)
              {
                  Console.WriteLine("Worker opened for {0}, {1}", name, context.PartitionKeyRangeId);
                  return Task.CompletedTask;  // Requires targeting .NET 4.6+.
              }
              public Task CloseAsync(ChangeFeedObserverContext context, ChangeFeedObserverCloseReason reason)
              {
                  Console.WriteLine("Worker closed for  {0}, {1}", name, context.PartitionKeyRangeId);
                  return Task.CompletedTask;
              }
  
              public Task ProcessChangesAsync(ChangeFeedObserverContext context, IReadOnlyList&lt;Microsoft.Azure.Documents.Document&gt; docs)
              {
                  Console.WriteLine("Change feed in {0}: total {1} doc(s)", name, Interlocked.Add(ref s_totalDocs, docs.Count));
                  foreach (var one in docs)
                  {
                      Console.WriteLine("Replicated: " + one.Id);
                      Replicate(one);
                  }
                  return Task.CompletedTask;
              }
          }
      }
  </code></pre>
  
  <p>And initialization of Observer:</p>
  
  <pre><code>        public async void StartReplication(string replicateFrom, string hostName, Func&lt;dynamic, bool&gt; replicate)
          {
  
              // Customizable change feed option and host options 
              ChangeFeedOptions feedOptions = new ChangeFeedOptions();
  
              // ie customize StartFromBeginning so change feed reads from beginning
              // can customize MaxItemCount, PartitonKeyRangeId, RequestContinuation, SessionToken and StartFromBeginning
              // feedOptions.StartFromBeginning = true;
  
              ChangeFeedHostOptions feedHostOptions = new ChangeFeedHostOptions();
  
              // ie. customizing lease renewal interval to 15 seconds
              // can customize LeaseRenewInterval, LeaseAcquireInterval, LeaseExpirationInterval, FeedPollDelay 
              //feedHostOptions.LeaseRenewInterval = TimeSpan.FromSeconds(15);
              feedHostOptions.LeasePrefix = replicateFrom;
              feedHostOptions.FeedPollDelay = TimeSpan.FromSeconds(60);
  
              try
              {
                  DocumentCollectionInfo documentCollectionLocation = new DocumentCollectionInfo
                  {
                      Uri = new Uri(_databaseServiceUri),
                      MasterKey = _databaseAuthKey,
                      DatabaseName = _databaseId,
                      CollectionName = replicateFrom
                  };
                  DocumentCollectionInfo leaseCollectionLocation = new DocumentCollectionInfo
                  {
                      Uri = new Uri(_databaseServiceUri),
                      MasterKey = _databaseAuthKey,
                      DatabaseName = _databaseId,
                      CollectionName = "leases"
                  };
                  DocumentFeedObserverFactory docObserverFactory = new DocumentFeedObserverFactory(replicate, replicateFrom);
                  ChangeFeedEventHost host = new ChangeFeedEventHost(hostName, documentCollectionLocation, leaseCollectionLocation, feedOptions, feedHostOptions);
  
                  await host.RegisterObserverFactoryAsync(docObserverFactory);
                  hosts.Add(host);
              }
              catch (Exception e)
              {
                  Console.WriteLine("Replication process halted for (" + replicateFrom + ") reason: " + e.ToString());
                  //throw e;
              }
          }
  </code></pre>