title: Asynchronous tests execute methods in other tests after their own tests have finished
body: <p>When I execute my tests one by one, all tests work and there are no problems. However when I execute them all together, I will run into problems like <code>NullReferenceException</code> and <code>SemaphoreFullException</code>.</p>
  
  <p>After a lot of digging I have noticed that the threads of my tests seem to interfere with eachother: even after the test has finished, (duplicated?) background calls seem to continue when another test is being executed.</p>
  
  <p>The application in question is a WinRT Component executed from a Universal App Unit Test Project. All calls you'll see below are executed asynchronously. In order to give my backgroundtasks the time to finish (if the test continues while supposedly waiting for the asynchronous background task, it will shutdown once it reaches the end of the test) I put the main thread to sleep for an appropriate amount of time with </p>
  
  <pre><code>new ManualResetEvent(false).WaitOne(millisecondsTimeout: 60 * 1000);
  </code></pre>
  
  <p>Additionally I have also added a sleep to the teardown method to be very sure everything gets finished:</p>
  
  <pre><code>[TestCleanup]
  public void Cleanup()
  {
     new ManualResetEvent(false).WaitOne(millisecondsTimeout: 60 * 1000);
  }
  </code></pre>
  
  <p>In order to diagnose what I have so far, I have looked at the execution plan of the tests and since they're pretty similar I simply put a <code>Debug.WriteLine(session.GetHashCode())</code> in every relevant place where my code passes. Below you can find the result of that stripped from the <code>FileNotFoundExceptions</code> in mscorlib, end-of-thread messages and symbol loading messages.</p>
  
  <p>This <code>Session</code> object in question is something that gets created by the user and passes it on to <code>Core</code> which then passes it on to <code>MainApi</code> and <code>StorageApi</code>. The former wraps this into a <code>Dispatcher</code> and passes this dispatcher onto <code>UserController</code> while <code>StorageApi</code> simply passes the session object to <code>StorageController</code> and subsequently <code>DatabaseController</code>.</p>
  
  <p>Throughout the entire hierarchy the same session object is used and passed through and all objects use the reference that is available to them at their location.</p>
  
  <p>The semaphore comes into play when a database is created. In the <code>EventBus</code> there is a property </p>
  
  <pre><code>internal static SemaphoreSlim TablesCreatedSemaphore = new SemaphoreSlim(0, 1);
  </code></pre>
  
  <p>which is used to make sure that there will be no data requested before the tables are created. This is used by placing </p>
  
  <pre><code>await EventBus.TablesCreatedSemaphore.WaitAsync();
  </code></pre>
  
  <p>between executing the <code>SuccesfulLogin</code> event and requesting data from the API and placing</p>
  
  <pre><code>EventBus.TablesCreatedSemaphore.Release();
  </code></pre>
  
  <p>Right after where the last table is created.</p>
  
  <pre><code>Integration_DownloadAsset_WithInvalidId_ThrowsSomeException
  *************************************************
                     NEW EXECUTION
  *************************************************
  Core (ctor): 53578018
  Main Api (ctor): 53578018
  Api dispatcher (ctor): 53578018
  UserController (ctor): 53578018
  Storage Api (ctor): 53578018
  Storage controller (ctor): 53578018
  Database controller (ctor): 53578018
  User controller (GetApiKeyAsync:pre-call): 53578018
  Session (IsLoggedIn): 53578018
  User controller (GetApiKeyAsync:pre-successfullogin-event): 53578018
  Storage controller (CreateFolderStructure): 53578018
  Database Controller(CreateDatabase): 53578018
  Session (IsLoggedIn): 53578018
  Session (IsLoggedIn): 53578018
  Database controller (ctor): 53578018
  
  
  
  Integration_DownloadAsset_WhenFileAlreadyExists_IsLocalReturnsTrue
  A first chance exception of type 'System.Exception' occurred in mscorlib.dll
  
  
  Integration_Login_WithValidLogin_RemovesUnusedAssetsFromFolder
  A first chance exception of type 'System.Exception' occurred in mscorlib.dll
  
  Integration_DownloadAsset_WithValidId_DownloadsFileToLocalFolder
  *************************************************
                     NEW EXECUTION
  *************************************************
  Core (ctor): 20039337
  Main Api (ctor): 20039337
  Api dispatcher (ctor): 20039337
  UserController (ctor): 20039337
  Storage Api (ctor): 20039337
  Storage controller (ctor): 20039337
  Database controller (ctor): 20039337
  User controller (GetApiKeyAsync:pre-call): 20039337
  Session (IsLoggedIn): 20039337
  User controller (GetApiKeyAsync:pre-successfullogin-event): 20039337
  Storage controller (CreateFolderStructure): 53578018
  Storage controller (CreateFolderStructure): 20039337
  Database Controller(CreateDatabase): 53578018
  Database Controller(CreateDatabase): 20039337
  Session (IsLoggedIn): 20039337
  Session (IsLoggedIn): 20039337
  Database controller (ctor): 20039337
  
  *************************************************
                     NEW EXECUTION
  *************************************************
  Core (ctor): 20995649
  Main Api (ctor): 20995649
  Api dispatcher (ctor): 20995649
  UserController (ctor): 20995649
  Storage Api (ctor): 20995649
  Storage controller (ctor): 20995649
  Database controller (ctor): 20995649
  User controller (GetApiKeyAsync:pre-call): 20995649
  Session (IsLoggedIn): 20995649
  User controller (GetApiKeyAsync:pre-successfullogin-event): 20995649
  Storage controller (CreateFolderStructure): 53578018
  Storage controller (CreateFolderStructure): 20039337
  Database Controller(CreateDatabase): 53578018
  Database Controller(CreateDatabase): 20039337
  Storage controller (CreateFolderStructure): 20995649
  Database Controller(CreateDatabase): 20995649
  Session (IsLoggedIn): 20995649
  A first chance exception of type 'System.Threading.SemaphoreFullException' occurred in mscorlib.dll
  A first chance exception of type 'System.Threading.SemaphoreFullException' occurred in mscorlib.dll
  </code></pre>
  
  <h1>Notes</h1>
  
  <ul>
  <li><p>Each test shows its name by printing it to <code>Debug.WriteLine</code> but apparently the last one didn't do this.</p></li>
  <li><p>The two tests with an <code>Exception</code> are failed tests and might very well be correct exceptions.</p></li>
  <li><p>The first test only has hashcodes of the object it created, the second one has calls to its methods with an object that has the hashcode as the previous test and the last test has calls to its methods with hashcodes from both previous tests.</p></li>
  <li><p>These findings and my assumption fall in line with some other curious behaviour I have noticed (sometimes the <code>Session</code> object wouldn't have authentication information in one place but do have it elsewhere, or data would get inserted inside the database twice despite primary key conflicts - although I don't know why that even was possible in the first place).</p></li>
  <li><p>I have considered that might just be an issue of late-flushing the data to the output window but it seems a little to well-timed for that.</p></li>
  <li><p>Changing the execution order of the tests still causes the first two to succeed and the third one to fail. The behaviour is the same with every run: each test performs an extra call with the session object of each previous test.</p></li>
  </ul>
  
  <h1>Example test</h1>
  
  <pre><code>[TestMethod]
  public async Task Integration_DownloadAsset_WithValidId_DownloadsFileToLocalFolder()
  {
    Debug.WriteLine("Integration_DownloadAsset_WithValidId_DownloadsFileToLocalFolder");
    var core = new Core(new Session());
    await core.Api.GetApiKeyAsync(new GetApiKeyRequestParameters
    {
        // Authentication parameters
    });
  
    //TODO: why is this necessary?
    new ManualResetEvent(false).WaitOne(millisecondsTimeout: 60 * 1000);
  
    await core.Api.DownloadAssetAsync("1891");
  
    // Allow the program to download the file
    new ManualResetEvent(false).WaitOne(millisecondsTimeout: 60 * 1000);
  
    var localFolder = await (await (await ApplicationData.Current.LocalFolder.GetFolderAsync("Data")).GetFolderAsync("AccountData")).GetFolderAsync("Files");
    var fileNames = (await localFolder.GetFilesAsync()).Select(x =&gt; x.Name).ToList();
    CollectionAssert.Contains(fileNames, "1891");
  }
  </code></pre>
  
  <h1>Question</h1>
  
  <p>I believe the cause to be that method invocations somehow remain behind even when the test has finished. How is this possible and how can I prevent from tests to intertwine with eachother?</p>
  
  <p>If this is not the cause, what is?</p>