Is the memory reported in Activity Monitor accurate?
You are doing the calculations wrong. In the column you see: physical memory, memory used, cached files, and swap used.
Physical memory is total RAM. Memory used is amount of RAM used for running processes. Cached files is the amount of RAM used for storing cache files, and swap used is the amount of disk (HDD/SSD) space that is being used as though it is RAM. So the calculation should be: Physical memory - (memory used + cached files). In your case: 4 GB - (2.91 + 1.07) (give or take).

However, what's really important is memory pressure. The system is good about using the RAM wisely. Your performance shouldn't be impeded as long as the memory pressure stays in the green.