Reduce SQL Server Memory used by a Database
Your best option is going to be using Resource Governor. Just create a resource for that vendor app and throttle the memory down for it.

You just have to come up with what you are going to identify that workload with, you are allowed:


The ability to classify incoming connections and route their
workloads to a specific group.
The ability to monitor resource usage for each workload in a group.
The ability to pool resources and set pool-specific limits on CPU
usage and memory allocation. This prevents or minimizes the
probability of run-away queries.
The ability to associate grouped workloads with a specific pool of
resources.
The ability to identify and set priorities for workloads.