1. A method comprising: detecting a page fault associated with a request for a device assigned to a guest operating system to perform direct memory access (DMA) of a requested page of memory; invalidating a mapping in a central processing unit (CPU) page table of a guest physical address to a host physical address for a candidate page for being swapped out of host memory; checking, by a hypervisor executing on a processing device, a DMA access state of the candidate page to determine whether or not the candidate page can be swapped out from the host memory; and removing the candidate page from the host memory in response to determining that the DMA access state for the candidate page indicates that the candidate page can be swapped out.