postgresql pg_dump out of memory
What the graph shows is pg_dump probably starting at 19:45 and having a normal memory consumption until 20:00. At 20:00, it appears that ~30 GB are suddenly allocated and not much is released until 23:15 when presumably the Out-Of-Memory condition occurs. 

From the comments, the kernel says:

hive-psql kernel: [15614.408169] Killed process 2693 (pg_dump) total-vm:38363104kB, anon-rss:34020140kB, file-rss:24kB, shmem-rss:0kB


The anon-rss of pg_dump at about 36GB shows that it is indeed pg_dump that over-allocates. There is something wrong in pg_dump needing that much memory.

Your database surely has many large objects, some of them may be very large (the theorical maximum per object is 2TB) but pg_dump processes large objects by:


opening a cursor on SELECT oid FROM pg_largeobject_metadata ORDER BY 1 and reading it by chunks of 1000 entries: that should never blow up.
for each large object, loop on lo_read to fetch the contents into a buffer of 16384 bytes: that should never blow up.


The fact that the table itself weighs 390 GB should not be a reason for pg_dump to do large allocations.


  But, first time pg_dump was killed at "retreiveing large objects"


Assuming pg_dump displays this with the -v options (by the way, you did not mention what options were passed), it should display "reading large objects" instead.