title: Cant find element in set (Psycopg, PostgreSQL)
body: <p>This should be easy, so maybe I've fried my brain, but I can't get this to work. I have a database with over a million rows. I want to add about 15k more rows, but some of the data might already exist in the database.</p>
  <p>My basic process is this:</p>
  <pre><code>async def process_data(object_list):
      # Load pre-existing data from db:
      pks = await copy_pks()
      # Remove duplicates:
      filtered_data = [obj for obj in object_list if obj.id not in pks]
      # Add unique values:
      await copy_in(filtered_data)
  </code></pre>
  <p>My <code>copy_pks</code> method:</p>
  <pre><code>@with_connection:
  async def copy_pks(self, connection):
      pks = set()
      async with connection.cursor().copy(sql) as cursor:
          async for row in cursor:
              res = await cursor.read_row()
              pks.add(int(res[0]))
      return pks
  </code></pre>
  <p>My <code>copy_in</code> method:</p>
  <pre><code>@with_connection
  async def copy_in(self, connection, data):
      async with connection.cursor().copy(sql) as cursor:
          for obj in data:
              await cursor.write_row(data)
  </code></pre>
  <p>The problem is that I keep hitting <code>duplicate key value violates unique constraint</code> despite my attempt to filter. I've checked the length of <code>object_list</code> and the length of <code>filtered_data</code> and the latter is about half the size... so clearly something is getting filtered.</p>
  <p>I then checked specifically for the offending object:</p>
  <pre><code>for obj in object_list:
      if obj.id == 14856714938:
          print(&quot;Found&quot;)
          if obj.id in pks:
              print(&quot;In pks&quot;)
          if obj.id in filtered_data:
              print(&quot;In filtered&quot;)
  </code></pre>
  <p>I receive &quot;Found&quot;, but I don't receive &quot;In pks&quot; and I don't receive &quot;In filtered.&quot;</p>
  <p>Unless I'm blind to some syntax or logic error here, my best shot in the dark is that it's a RAM issue, given the size of the data?</p>