Optimize INSERT performance with a UNIQUE constraint on md5(JSONB_COLUMN::text) I have the following table and indices defined:

CREATE TABLE IF NOT EXISTS data (
    id serial,
    job_id bigint NOT NULL,
    payload jsonb NOT NULL,
    tags jsonb NULL
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_data_content on data (md5(payload::text));
CREATE INDEX IF NOT EXISTS idx_data_tags ON data USING GIN (tags jsonb_path_ops);
CREATE INDEX IF NOT EXISTS idx_data_jid ON data USING btree (job_id);


In that table, I store many JSON blobs, which are associated with a job_id (also available in payload-&gt;&gt;'jobId') and a time stamp (unix epoch) stored in payload-&gt;&gt;'__time'. I want to avoid storing duplicate JSON blobs, so I'm using this method as a UNIQUE constraint. After a few hundred thousand rows in the DB, however, INSERT performance starts to decline, which does not happen without the UNIQUE constraint.

It is highly unlikely but possible to have two different JSON blobs with the same job_id and payload-&gt;&gt;'__time'. Is there any way I can make use of that, for example, by defining a UNIQUE constraint that first checks the job_id, then payload-&gt;&gt;'__time', and only if both match actually checks the md5(payload::text) index? 

(Note, that the keys in my JSON blobs are always in the same order for the same documents.)
If the uniqueness constraint is the only issue (and I'm interested to learn more about why in the discussion above), here's an idea: 


remove the uniqueness constraint
when you do reads (selects), do order by id asc limit 1 so that you ignore duplicates
have some sort of parallel process going through the table periodically and removing duplicates