title: How do I find all the distinct keys of json records in c#?
body: <p>Why this question is <strong>not duplicate</strong>? (Added after seeing comments)</p>
  
  <ul>
  <li>It is not related to Entity framework. </li>
  <li>It has to deal with parsing huge json files and find distinct keys, but not records!</li>
  </ul>
  
  <p>I've 200+ files and each of them is 2+ GB, implies total size is 400+ GB. Each line in those files is a json string. I don't have <code>json schema</code> for the records beforehand. My job is to find all the keys in those files. </p>
  
  <p>I wrote following code to get all the distinct keys from all those json records. I call following method using multi-threaded <code>for-loop</code> from <code>main</code>.</p>
  
  <pre><code>private void GetTokensFromJson(string filePath)
          {
              IEnumerable&lt;string&gt; txts = File.ReadLines(filePath, Encoding.UTF8);
  
              Console.WriteLine(txts.Count());
  
              List&lt;string&gt; distinctKeys = new List&lt;string&gt;();
  
              foreach (var text in txts)
              {
  
                      string pattern = "{\"";
  
                      foreach (Match m in Regex.Matches(text, pattern))
                      {
                          //string matchValue = m.Value;
                          int matchIndex = m.Index;
                          string subStr=text.Substring(matchIndex+2, text.Length - matchIndex - 3);
                          int quoteIndex=subStr.IndexOf('\"');
                          string jsonKey = subStr.Substring(0, quoteIndex);
                          if (!distinctKeys.Contains(jsonKey) &amp;&amp; !jsonKey.Contains("\\"))
                          {
                              Console.WriteLine(jsonKey);
                              distinctKeys.Add(jsonKey);
                          }
                      }
  
                  string secondPattern="\":";
                  foreach (Match m in Regex.Matches(text, secondPattern))
                  {
                      int matchIndex = m.Index;
                      string revJsonKKey = "";
                      while(matchIndex&gt;0)
                      {
                          matchIndex--;
                          if (text[matchIndex] != '\"')
                              revJsonKKey += text[matchIndex];
                          else
                              break;
                      }
  
                      IEnumerable&lt;char&gt; jsonKeyCharArray = revJsonKKey.Reverse();
                      string jsonKey="";
                      foreach(char c in jsonKeyCharArray)
                      {
                          jsonKey += c;
                      }
  
                      if (!distinctKeys.Contains(jsonKey) &amp;&amp; !jsonKey.Contains("\\"))
                      {
                          Console.WriteLine(jsonKey);
                          distinctKeys.Add(jsonKey);
                      }
  
                  }
  
              }
  </code></pre>
  
  <p><code>distinctKeys</code> has all the distinct json keys. But I'm missing few keys and adding unwanted keys, not sure why :|. I can't debug for the given input, as it is too huge! Also, this method is too slow.</p>
  
  <p>To make things clearer, let's take an example, if files have following json,</p>
  
  <pre><code>{"id":"123", "name":"hello, world", "department":[{"name":"dept1", "deptID":"123"}]}
  {"id":"456324", "department":[{"name":"dept2", "deptID":"456"}]}
  </code></pre>
  
  <p>Expected <strong>output</strong> is <code>id,name,department, department-&gt;name, department-&gt;deptID</code>. Formatting of output doesn't matter. Note that not all the json records will not have all the keys and json record can contain nested json records.</p>
  
  <p>I've <strong>two</strong> questions,</p>
  
  <ol>
  <li>What am I doing wrong in the code?</li>
  <li>Is there a inbuilt or 3rd-party dll which will give me output as keys of json, when I give input as complex json record?</li>
  </ol>