How to check if memory usage goes above a certain number?
Assuming you have some linux knowledge, your question can be rephrased to this: https://stackoverflow.com/questions/131303/how-to-measure-actual-memory-usage-of-an-application-or-process 

Add a process memory check to a scheduled task that runs every x hours/minutes. If it exceeds a number kill the process.

Although, if you are confident the issue is with the application you are using, I would take it to their support team, or change the software with a better one.