Why are certain objects loaded into Buffer?
First, check the plan cache with queries that hit sys.dm_exec_query_stats. My favorite is the open source sp_BlitzCache (disclaimer: I'm one of the coauthors). Queries could be running outside of your control, like monitoring systems. I've seen third party tools that do things like check table fragmentation.

Next, not everything stays in the plan cache. Unfortunately - especially when you're dealing with monitoring software - some queries are smart enough to do a WITH (RECOMPILE) in order to avoid going into the plan cache. (sp_BlitzCache even does this.) To catch diabolical queries like this, consider doing a Profiler trace or Extended Events capture - although I wouldn't recommend leaving those in place long term. Only do it to satisfy your curiosity.