How to fix high memory usage of MariaDB?
It's not critical So don't worry It's a normal Behaviour of Innodb Engine. 

If your data size is bigger than or equal the Buffer pool size, Innodb will fill all of buffer size at warm up time. So when you start MySQL after some hours buffer pool will full (Only Buffer pull Not all Ram's capacity)

You'are using multiply Engines in your system. It can complicate config, because you have to controll important variables (such as Memory Configs) of each Engine.

Do you have a reason to use the TokuDB? I think it's better to use one of these engines (InnoDB or TokuDB)

So if you prefer to use innodb, you can set buffer pool size to about 70% of RAM unless your data/index size is small