UPDATE on big table in PostgreSQL randomly takes too long
GIN indexes have a &quot;fastupdate&quot; mechanism where new data gets written to a specific section in a linear fashion.  Once that exceeds some set size (gin_pending_list_limit, which is set globally but can be overridden per index) the next process which happens along to write to the index gets assigned the task of consolidating those entries into the main part of the index, which can lead to long freezes for that process.  For the length of freezing you are seeing, you must have a pretty high setting for gin_pending_list_limit.
If consistent latency is more important to you than overall insert/update performance, you can disable fastupdate.  Or you could lower the value of gin_pending_list_limit to keep some of the benefit while lowering the freeze-up time.
If you need both consistency and overall efficiency, you could write a manual process/thread which will open its own connection and run select gin_clean_pending_list(...) on the index in a loop while the main update operation is ongoing.  That way you get the benefit of the pending list, while the latency of cleaning it up is all loaded onto a background process where it doesn't matter.  (Vacuum and autovacuum will also clean it up, but you can't really get them to run often enough to do a sufficient job of it, because of all the other work they also need to do.  Hence gin_clean_pending_list)