1. A processor comprising: page walk hardware to traverse a hierarchy of page tables to perform an address translation of an actual linear address to a physical address, wherein the actual linear address is used by software to access a memory and the physical address is used by the processor to access the memory; a translation lookaside buffer to store a plurality of address translations performed by the page walk hardware; and memory access control hardware to look up, in a permissions map separate from the hierarchy of page tables, the physical address from the page walk hardware to find access permissions and an expected linear address, compare the expected linear address to the actual linear address, and store the address translation and the access permissions in the translation lookaside buffer only if the expected linear address matches the actual linear address.