PostgreSQL LWLock: lock_manager issue
The full object locks which last (usually) for the duration of a transaction and which you can see in pg_locks have info about them stored in shared memory.  But access to that shared memory requires the protection of light-weight locks, which should last for only nanoseconds or microseconds while the memory access is actually occuring.  What you appear to have is contention on those light-weight locks which protect the memory used for the full locks.
I tried to reproduce this contention by having intense traffic through the lock manager, but I can't reproduce this contention.  So I think you are doing something which is not obvious based your description, like maybe using subtransactions.  Could you publish code which does reproduce this problem, by inserting, selecting, and updating the way you do it, but with the rest of your business logic stripped out?
But in any case, allowing there to be over 200 active connections in a machine with only 4 CPUs (for EC2 x86 machines, the number of usable CPU is generally half the number of vCPU) can't be good for contention.  The more active processes you have, the more time they spend fighting over locks rather than doing useful work.  Having a lot of connections can simplify the client book-keeping and it isn't always a problem (like if few are active at any one time), but there is ample evidence here that for you it is a problem.
You should adjust your connection pooler to restrict the number of connections.  If 90 of your 100 connections are usually idle, then they are not useful during the time when they are idle, and are actually harmful during the time they are in use as they cause contention. This creates a &quot;the beatings shall continue until morale improves&quot; situation--the database is slow so let's launch more connections, which create more contention which slow it further. (They are kind of harmful all the time, as they impede the usefulness of your Connections Count monitoring graph). Are the consumers connecting through the same pooler as the producers?