title: C# - fastest way of comparing a collection against itself to find duplicates
body: <pre><code>public class TestObject
  {
      string TestValue { get; set; }
      bool IsDuplicate { get; set; }
  }
  
  List&lt;TestObject&gt; testList = new List&lt;TestObject&gt;
  {
      new TestObject { TestValue = "Matt" },
      new TestObject { TestValue = "Bob" },
      new TestObject { TestValue = "Alice" },
      new TestObject { TestValue = "Matt" },
      new TestObject { TestValue = "Claire" },
      new TestObject { TestValue = "Matt" }
  };
  </code></pre>
  
  <p>Imagine <code>testList</code> is actually millions of objects long.</p>
  
  <p>What's the fastest way to ensure that two of those three <code>TestObjects</code> with <code>TestValue</code> of Matt gets its <code>IsDuplicate</code> set to true? No matter how may instances of a given value there are, only one should come out of the process with IsDuplicate of false.</p>
  
  <p>I am not averse to doing this via threading. And the collection doesn't have to be a list if converting it to another collection type is faster.</p>
  
  <p>I need to keep duplicates and mark them as such, not remove them from the collection.</p>
  
  <p>To expand, this is (as you might imagine) a simple expression of a much more complex problem. The objects in question already have an ordinal which I can use to order them. </p>
  
  <p>After matching initial duplicates on exact string equality, I'm going to have to go back through the collection again and re-try the remainder using some fuzzy matching logic. The collection that exists at the start of this process won't be changed during the deduplication, or afterwards.</p>
  
  <p>Eventually the original collection is going to be written out to a file, with likely duplicates flagged.</p>