Mysql consumes a lot of memory Here is my situation.

I have an ubuntu server monitored by nagios.
recently nagios sends alarms regarding the memory consumption on the server. I logged into the server and executed the top command then I found that mysql is the process that consumes a lot of the memory (about 64%).

This server contains a forum, and DB replica for another server so I logged into mysql and show processlist I found only 7 records.

How could I dig more in this situation to decide what is the application which consumes all that memory?
MySQL consumes 64% is not a very big deal if the load is very high and the memory is low on the machine. Typically I would say its better that it should not consume more than 20 to 25% of the total memory available.

Anyways for your question, just try to find out the bottlenecks. 
1 You should be able to find out if there are any long running queries - Say for example you are fetching some 1 million rows or so in a InnoDB engine (where transaction is supported) and your query has lot of joins.
2 Then you can tweak the actual config file of MySQL to increase the number of connections that is allowed and the max memory that is assigned to MySQL.
3 Also make sure you close all the sql connections in your application if you have not done it.
4 If long running queries are the bottleneck, then you can change some of the tables to MyIsam if it doesn't matter with transactions.

I hope it helps !