1. An apparatus comprising: an accelerator comprising one or more execution units (EUs) to execute a specified set of instructions; a front end core to the accelerator comprising: a translation lookaside buffer (TLB) communicatively coupled to and shared with the accelerator and providing memory access services to the accelerator, the memory access services including performing TLB lookup operations to map virtual to physical addresses on behalf of the accelerator in response to the accelerator requiring access to a system memory, and a TLB shootdown handler for performing a TLB shootdown operation at the TLB of the front end core, in response to detecting that a particular page has been restricted to use by a particular system component, the TLB shootdown operation causing TLB entries for the particular page to be flushed; and a memory hierarchy including the system memory communicatively coupled to both the front end core and the accelerator and accessible via the physical addresses provided by the TLB lookup operations.