title: How do I systematically compare all rows in two Pandas dataframes using specific columns and return the differences?
body: <p>I have two large dataframes from different sources, largely of the same structure but of different lengths and in a different order. Most of the data is comparable but not all. The rows represent individuals and the the columns contain data about those individuals. I want to check by row certain column values of one dataframe against the 'master' dataframe and then return the omissions so these can be added to it.</p>
  
  <p>I have been using the <code>df.query()</code> method to check individual cases using my own inputs because I can search the master dataframe using multiple columns - so, something like <code>df.query('surname == "JONES" and initials == "D V" and town == "LONDON"')</code>. I want to do something like this but by creating a query of each row of the other dataframe using data from specific columns.</p>
  
  <p>I think I can work out how I might do this using for loops and if statements but that's going to be wildly inefficient and <a href="https://stackoverflow.com/questions/16476924/how-to-iterate-over-rows-in-a-dataframe-in-pandas/55557758#55557758">obviously not ideal</a>. List comprehension might be more efficient but I can't work out the dataframe comparison part unless I create a new column whose data is built from the values I want to compare (JONES-DV-LONDON, but that seems wrong).</p>
  
  <p>There is an answer <a href="https://hackersandslackers.com/compare-rows-pandas-dataframes/" rel="nofollow noreferrer">in here</a> I think but it relies on the dataframes being more or less identical (which mine aren't - hence my wish to compare only certain columns).</p>
  
  <p>I have been unable to find an example of someone doing the same, which might be my failure again. I am a novice and I have a feeling I might be thinking about this in completely the wrong way. I would very much value any thoughts and pointers... </p>
  
  <p><strong>EDIT</strong> - some sample data (not exactly what I'm using but hopefully helps show what I am trying to do)</p>
  
  <pre><code>df1 (my master list)
  surname    initials    town
  JONES      D V         LONDON
  DAVIES     H G         BIRMINGHAM
  
  df2
  surname    initials    town
  DAVIES     H G         BIRMINGHAM
  HARRIS     P J         SOUTHAMPTON
  JONES      D V         LONDON
  </code></pre>
  
  <p>I would like to identify the columns to use in the comparison (surname, initials, town here - assume there are more with data that cannot be matched) and then return the unique results from df2 - so in this case a dataframe containing:</p>
  
  <pre><code>surname    initials    town
  HARRIS     P J         SOUTHAMPTON
  </code></pre>