Processing lots of JSON objects I want to process a ton of JSON objects as quickly as possible.

I've done my best to make the processing as concise and efficient as I can.

How can I make it more efficient?

private void processButton_Click(object sender, EventArgs e)
{
    // For measuring how long the processing takes.
    var stopwatch = Stopwatch.StartNew();

    // Get the IDs of the completed RootObjects.
    // There are no duplicate IDs.
    var completed = new List&lt;string&gt;();
    foreach (var file in Directory.EnumerateFiles("C:\\Completed", "*.json"))
    {
        completed.AddRange(
            JsonConvert.DeserializeObject&lt;List&lt;RootObject&gt;&gt;(File.ReadAllText(file)).Select(o =&gt; o.id));
    }
    Console.WriteLine($"completed.Count: {completed.Count}");

    // Get the unfinished RootObjects.
    //
    // 78,198 of the unfinished RootObjects share their ID with another.
    // The duplicates are removed in the next step. (#2)
    //
    // The unfinished RootObjects also contain ALL of the completed RootObjects.
    // The completed RootObjects are ignored in the next step. (#1)
    var unfinished = new List&lt;RootObject&gt;();
    foreach (var file in Directory.EnumerateFiles("C:\\Unfinished", "*.json"))
    {
        unfinished.AddRange(JsonConvert.DeserializeObject&lt;List&lt;RootObject&gt;&gt;(File.ReadAllText(file)));
    }
    Console.WriteLine($"unfinished.Count: {unfinished.Count}");

    var processed =
        unfinished.Where(o =&gt; !completed.Contains(o.id)) // (#1) Ignore all completed RootObjects.
            .GroupBy(o =&gt; o.id).Select(objects =&gt; objects.First()) // (#2) Remove all duplicate RootObjects.
            .ToList();
    Console.WriteLine($"processed.Count: {processed.Count}");

    stopwatch.Stop();
    Console.WriteLine($"stopwatch.ElapsedMilliseconds: {stopwatch.ElapsedMilliseconds}");
}

// Output:
//      complete.Count: 35649
//      unfinished.Count: 250315
//      processed.Count: 136468
//      stopwatch.ElapsedMilliseconds: 75875


If you require any further information, let me know.
LINQ (convenient)

You could rewrite the loops with LINQ so this


var completed = new List&lt;string&gt;();
foreach (var file in Directory.EnumerateFiles("C:\\Completed", "*.json"))
{
  completed.AddRange(
      JsonConvert.DeserializeObject&lt;List&lt;RootObject&gt;&gt;(File.ReadAllText(file)).Select(o =&gt; o.id));
}



could become this

var completd = 
    Directory.EnumerateFiles("C:\\Completed", "*.json")
    .Select(File.ReadAllText)
    .Select(json =&gt; JsonConvert.DeserializeObject&lt;List&lt;RootObject&gt;&gt;(json).Select(x =&gt; x.id))
    .SelectMany(x =&gt; x)
    .ToList();


You can do the same with the other loop.

When using LINQ you might experience some performance hit but don't be discouraged from using it. Use LINQ first because it's easy to use and read. Only if you are not satisfied with its performance rewrite it with a normal loop.

In most cases you won't even notice that LINQ is in action.

Run a profiler if you're not sure. Don't blindly rewrite everything with loops because LINQ potentialiy could slow something down unless you really, really, really need to.




var processed =
  unfinished.Where(o =&gt; !completed.Contains(o.id)) // (#1) Ignore all completed RootObjects.
      .GroupBy(o =&gt; o.id).Select(objects =&gt; objects.First()) // (#2) Remove all duplicate RootObjects.
      .ToList();



I think this is the most inefficient part in your code because the completed.Contains method is a O(n) operation.

There are two alternatives for it: either use a HashSets or the Except extension that


  [..] returns those elements in first that do not appear in second. It does not also return those elements in second that do not appear in first.


In order to make it work and make its usage easier you should get entire RootObjects from the first query instead of just the ids and you'll need a custom comparer for the RootObject that you can even reuse for the Distinct extension so you can replace the


.GroupBy(o =&gt; o.id).Select(objects =&gt; objects.First())



with 

.Distinct(new RootObjectComparer())


The new query is now much easier to understand and does not requrie any comments. It also swaps the Distinct with the Except so it has less items to process.

var processed =
    unfinished
    .Distinct(new RootObjectComparer())
    .Except(completed, new RootObjectComparer())
    .ToList();


The custom comparer.

class RootObjectComparer : IEqualityComparer&lt;RootObject&gt;
{ 
    public bool Equals(RootObject left, RootObject right)
    {
        return left.id == right.id;
    }

    public int GetHashCode(RootObject obj)
    {
        return obj.id;
    }
}




LINQ free (fast)

You could try to mix LINQ and the HashSet into this where you not only provide the root-objects to the constructor but also the custom comparer.

A hash-set stores only unique objects so there's no need to filter them.

var unfinished = new HashSet&lt;RootObject&gt;(
    Directory
        .EnumerateFiles("C:\\Unfinished", "*.json")
        .Select(File.ReadAllText)
        .Select(json =&gt; JsonConvert.DeserializeObject&lt;List&lt;RootObject&gt;&gt;(json))
        .SelectMany(x =&gt; x),
    new RootObjectComparer()
);




The fastet version should be however this LINQ free one. Here you import the lists directly into the hash-set.

var unfinished = new HashSet&lt;RootObject&gt;(new RootObjectComparer());

foreach (var file in Directory.EnumerateFiles("C:\\Unfinished", "*.json"))
{
    unfinished.UnionWith(
        JsonConvert.DeserializeObject&lt;List&lt;RootObject&gt;&gt;(File.ReadAllText(file)));
}


Then you exclude the completed items with

unfinished.ExceptWith(completed);