title: Intermittent "Owner resource does not exist"
body: <p>I am using DocumentDb v1.8.0 and when checking if a collection exists it will return a 404 error. If I get that error, I then try to create the collection and that returns a 409 error (resource already exists).</p>
  
  <p>I have four collections and one of them randomly fails every time this check is done. The databases/collections are not being deleted so I am at a loss as to why this is happening.</p>
  
  <p>some other information:</p>
  
  <ul>
  <li>The collections IndexingMode is consistent</li>
  <li>ConnectionMode is direct</li>
  <li>Protocol is TCP</li>
  </ul>
  
  <p>Update with stacktrace:</p>
  
  <pre><code>{"ClassName":"System.AggregateException","Message":"One or more errors occurred.","Data":null,"InnerException":{"ClassName":"Microsoft.Azure.Documents.DocumentClientException","Message":"Message: {\"Errors\":[\"Owner resource does not exist\"]}\r\nActivityId: 89bb346d-762d-430f-a1cb-dc4a3c55ac24, Request URI: /apps/bfb8961c-dcec-4c64-b341-400abf86ebdb/services/2ace1923-f76e-454f-9e2a-ea3a454dfb82/partitions/179a8dae-314e-4c15-878e-715cbb0e4f2e/replicas/131142481735385314s","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"   at Microsoft.Azure.Documents.Client.ClientExtensions.&lt;ParseResponseAsync&gt;d__4.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.Routing.GatewayAddressCache.&lt;ResolveAddressesViaGatewayAsync&gt;d__11.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.Routing.GatewayAddressCache.&lt;ResolveServerAsync&gt;d__a.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.Routing.GatewayAddressCache.&lt;ResolveAsync&gt;d__0.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.AddressSelector.&lt;ResolveAddressesAsync&gt;d__15.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.AddressSelector.&lt;ResolveAllUriAsync&gt;d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreReader.&lt;ReadMultipleReplicasInternalAsync&gt;d__a.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreReader.&lt;ReadMultipleReplicaAsync&gt;d__0.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyReader.&lt;ReadSessionAsync&gt;d__8.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.&lt;InvokeAsync&gt;d__b.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.&lt;&gt;c__DisplayClass1.&lt;&lt;InvokeAsync&gt;b__0&gt;d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.&lt;&gt;c__DisplayClassd`1.&lt;&lt;ExecuteAsync&gt;b__c&gt;d__f.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.&lt;ExecuteRetry&gt;d__16.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.&lt;ExecuteAsync&gt;d__12`1.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.&lt;InvokeAsync&gt;d__7.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.&lt;ProcessMessageAsync&gt;d__0.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.Client.DocumentClient.&lt;ExecuteQueryAsync&gt;d__2f0.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.Query.DocumentQueryClient.&lt;ExecuteQueryAsync&gt;d__4.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextBase.&lt;ExecuteQueryRequestAsync&gt;d__47.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext.&lt;ExecuteOnceAsync&gt;d__9.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.&lt;&gt;c__DisplayClass2.&lt;&lt;ExecuteAsync&gt;b__1&gt;d__4.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.&lt;ExecuteRetry&gt;d__16.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.&lt;ExecuteAsync&gt;d__7.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.Query.DefaultDocumentQueryExecutionContext.&lt;ExecuteInternalAsync&gt;d__6.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.Query.DocumentQueryExecutionContextBase.&lt;ExecuteNextAsync&gt;d__11.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.R
  </code></pre>
  
  <p>There are two instances of the IReliableReadWriteDocumentClient that are created in a static constructor provider class (one per Db) that lookup two collections.</p>
  
  <p>Update2: 
  Looking into it further, comparing the requests to the address routing table, the ReadCollection method (which 404s) targets the primary node, but the CreateCollection method (which 409s) is targeting a secondary node.
  Also I'm working a repro, it may take a little time unfortunately.</p>