slow wal replay in backup postgres12 server We have the following PG12 setup with Master, replica and readonly replica with archive. 


master and replica both has 2cpu, 4gb ram 
readonly replica with archive has 2 cpu, 2gb ram


below you can see that the db-app-2 and db-app-backup are both writing and flushing the WAL at the same time. But when it comes to WAL replay db-app-backup is far behind and the replay lag increases as the time goes.

CPU 69.2% RAM 34.0% HDD 61%

 application_name | client_addr  |   state   |   sent_lsn   |  write_lsn   |    write_lag    |  flush_lsn   |    flush_lag    |  replay_lsn  |   replay_lag    | sent_lag | write_lag | flush_lag | replay_lag
------------------+--------------+-----------+--------------+--------------+-----------------+--------------+-----------------+--------------+-----------------+----------+-----------+-----------+------------
 db-app-2         |   x.y.z.a    | streaming | 277/B8400000 | 277/B8400000 | 00:00:00.011132 | 277/B8400000 | 00:00:00.013937 | 277/B82EB168 | 00:00:00.213115 | 0 bytes  | 0 bytes   | 0 bytes   | 1108 kB
 db-app-backup    |   x.y.z.b    | streaming | 277/B8400000 | 277/B8400000 | 00:00:00.01765  | 277/B8400000 | 00:00:00.025976 | 274/C8DBC6E0 | 00:15:42.462623 | 0 bytes  | 0 bytes   | 0 bytes   | 12 GB


Any help in identifying the bottleneck for wal replay is appreciated.

Note:
The below query is used in master to fetch the data above

SELECT application_name, client_addr, state, sent_lsn, write_lsn, write_lag, flush_lsn, flush_lag, replay_lsn, replay_lag,
       pg_size_pretty(pg_wal_lsn_diff(
           pg_current_wal_lsn(),
           sent_lsn
       )) AS sent_lag,
       pg_size_pretty(pg_wal_lsn_diff(
           pg_current_wal_lsn(),
           write_lsn
       )) AS write_lag,
       pg_size_pretty(pg_wal_lsn_diff(
           pg_current_wal_lsn(),
           flush_lsn
       )) AS flush_lag,
       pg_size_pretty(pg_wal_lsn_diff(
           pg_current_wal_lsn(),
           replay_lsn
       )) AS replay_lag
  FROM pg_stat_replication;
There must be a replication conflict with a query that prevents applying the changes to the data files.

Look into pg_stat_database_conflicts on the standby for details.

You probably have max_standby_streaming_delay set to a value greater than 0 and long running queries on the standby. Now if some action on the primary server causes WAL entries whose application would conflict with the running query, replay of these data modifications gets delayed.

You can try to minimize conflicts by setting hot_standby_feedback to on (at the risk of bloating your primary server's tables), but you'll never totally get rid of them.

It is difficult to have a standby that is both never delayed and supports long running queries at the same time. Better use two standby servers with different max_standby_streaming_delay if you need both. Of course, if you want the standby for scaling reads, you want both at the same time, and you are in a difficult position. Streaming replication is not very good for that.

You could also experiment with setting vacuum_defer_cleanup_age to a value greater than 0 to reduce the number of replication conflicts caused by VACUUM.