title: Finding values and duplicates in large JSON dataset with python
body: <p>I have a huge data set (B) of JSON objects. I have another smaller data set (A) of JSON objects as well. What is the <strong>fastest</strong> way to see if every element in A is within B? Also how would I check if there are duplicates of elements in A that are in B?</p>
  
  <p>What I had in mind was creating a <code>dictionary</code> of the data set B with its key value pair being the same JSON value. This will allow for fast lookups. I will do the same thing for set A as well however the value for each key in set A will be an empty list. </p>
  
  <p>With each key in set A will be looked up on set B and appended to the respective list. Once this is done, the length of the list will determine if any of these values are not found, matched, duplicated.</p>
  
  <p>If the length of the list for a key in set A is </p>
  
  <pre><code>0  --&gt; none found in B
  1  --&gt; 1 found in B
  &gt;1 --&gt; more than one found in B (duplicates found)
  </code></pre>
  
  <p>I dont think standard dictionaries support duplicates, not really sure which data structure to use to support duplicate key, value pairs</p>