title: Detecting duplicates in pandas when a column contains lists
body: <p>Is there a reasonable way to detect duplicates in a Pandas dataframe when a column contains lists or numpy nd arrays, like the example below? I know I could convert the lists into strings, but the act of converting back and forth feels... wrong. Plus, lists seem more legible and convenient given <a href="https://www.online-python.com/jkprUEqnJi" rel="nofollow noreferrer">~how I got here (online code)</a> and where I'm going after.</p>
  <pre><code>import pandas as pd
  
  df = pd.DataFrame(
      {
          &quot;author&quot;: [&quot;Jefe98&quot;, &quot;Jefe98&quot;, &quot;Alex&quot;, &quot;Alex&quot;, &quot;Qbert&quot;],
          &quot;date&quot;: [1423112400, 1423112400, 1603112400, 1423115600, 1663526834],
          &quot;ingredients&quot;: [
              [&quot;ingredA&quot;, &quot;ingredB&quot;, &quot;ingredC&quot;],
              [&quot;ingredA&quot;, &quot;ingredB&quot;, &quot;ingredC&quot;],
              [&quot;ingredA&quot;, &quot;ingredB&quot;, &quot;ingredD&quot;],
              [&quot;ingredA&quot;, &quot;ingredB&quot;, &quot;ingredD&quot;, &quot;ingredE&quot;],
              [&quot;ingredB&quot;, &quot;ingredC&quot;, &quot;ingredF&quot;],
          ],
      }
  )
  
  # Traditional find duplicates
  # df[df.duplicated(keep=False)]
  
  # Avoiding pandas duplicated function (question 70596016 solution)
  i = [hash(tuple(i.values())) for i in df.to_dict(orient=&quot;records&quot;)]
  j = [i.count(k) &gt; 1 for k in i]
  df[j]
  </code></pre>
  <p>Both methods (the latter from <a href="https://stackoverflow.com/a/70596317/6417219">this alternative find duplicates answer</a>) result in</p>
  <blockquote>
  <p>TypeError: unhashable type: 'list'.</p>
  </blockquote>
  <p>They would work, of course, if the dataframe looked like this:</p>
  <pre><code>df = pd.DataFrame(
      {
          &quot;author&quot;: [&quot;Jefe98&quot;, &quot;Jefe98&quot;, &quot;Alex&quot;, &quot;Alex&quot;, &quot;Qbert&quot;],
          &quot;date&quot;: [1423112400, 1423112400, 1603112400, 1423115600, 1663526834],
          &quot;recipe&quot;: [
              &quot;recipeC&quot;,
              &quot;recipeC&quot;,
              &quot;recipeD&quot;,
              &quot;recipeE&quot;,
              &quot;recipeF&quot;,
          ],
      }
  )
  </code></pre>
  <p>Which made me wonder if something like integer encoding might be reasonable? It's not that different from converting to/from strings, but at least it's legible. Alternatively, suggestions for converting to a single string of ingredients per row directly from the starting dataframe in the <a href="https://www.online-python.com/jkprUEqnJi" rel="nofollow noreferrer">code link</a> above would be appreciated (i.e., avoiding lists altogether).</p>