Custom metric for a Windows process on a Windows ec2 instance
Memory consumption is a metric that has to be reported from inside the guest (EC2 instance).  You can use a CloudWatch agent to collect metrics from the guest directly. The agent also supports custom metrics now.

Alternatively, install a third party monitoring system. Which also means that you'll have to get an agent running on the EC2 instance. There are many of those, choose whatever fits your environment.