SQL Server 2012 running background task that fills memory
SQL Server does not use up memory without a workload running. I.E. an instance of SQL Server will consume the bare minimum it requires to keep the engine alive, listen for connections and handle background tasks (e.g. resource management). That's usually less than 100MB. If you have a min memory setting then it'll acquire that minimum IF there's demand because of workload and then sit there; won't release beyond the minimum even after the workload is done.

If your SQL Server process (sqlservr.exe in task manager or process monitor) is consuming lots of memory, something is going on. Check what's going on in the server using SELECT * FROM sys.dm_exec_sessions and look at the login_name column to see who's connected. That's a start to identify potential unknown workloads. From there, the other columns will provide information on how much resources it's using. It could be a real user, machine or even some old SQL Agent job gone wild.