Troubleshooting high memory usage on MariaDB 10.5.4
POSSIBLE MEMORY LEAK ISSUES
There have been three releases since version 10.5.4

03 Nov 2020 : MariaDB 10.5.7
07 Oct 2020 : MariaDB 10.5.6
10 Aug 2020 : MariaDB 10.5.5

There have been some memory issues fixed since version 10.5.4
Memory Issues Fixed in 10.5.5
Revision #d1bb7f9143

2020-06-24 12:51:55 +0300
Fixed memory leak in item_sum.cc::report_cut_value_error()

Revision #572e53d8cc

2020-06-19 15:24:16 +0530
MDEV-22931:  mtr_t::mtr_t() allocates some memory

Memory Issues Fixed in 10.5.6
None (just branch merges from previous versions)
Memory Issues Fixed in 10.5.7
Revision #0d581562d8

2020-10-22 17:09:18 +0300
Fix memory leak on Alter_drop allocation

Revision #d98ccbe1e1

2020-08-21 11:54:16 +0300
MDEV-23526 InnoDB leaks memory for some static objects

WHAT CAN YOU DO NEXT
You might need to update to MariaDB 10.5.7 on a Staging Server, load data into it.Once you do that, you can do some benchmarking.

Perhaps you can mysqlbinlog dump everything from the binaries logs in the 10.5.4 server and import the SQL generated into the Staging Server running 10.5.7.
You could also import the slow log from 10.5.4 server into 10.5.7 Staging Server.

See if memory bloats up or not.
If memory does not bloat, then you should upgrade the prod server to 10.5.7.