title: Spark: Find pairs having at least n common attributes?
body: <p>I have a dataset consisting of <code>(sensor_id, timestamp, data)</code> (the <code>sensor_id</code> are ids of IoT devices, timestamp is UNIX time and data is an MD5 hash of their output at that time). There is no primary key on the table but each row is unique.</p>
  
  <p>I need to find all pairs of <code>sensor_id</code>s <code>s1</code> and <code>s2</code> such that these two sensors have at least <code>n</code> (<code>n=50</code>) entries <code>(timestamp, data)</code> in common between them i.e. on <code>n</code> different occasions they emitted same data at same timestamp.</p>
  
  <p>For a sense of magnitudes of the data, I have 10B rows and ~50M distinct <code>sensor_ids</code> and I believe that there are around ~5M pairs of sensor-ids that emitted same data at same timestamp at least 50 times.</p>
  
  <p>What's the best way to do this in Spark? I tried various approaches (group-by <code>(timestamp, data)</code> and/or self-joining) but they are prohibitively expensive in complexity.</p>