1. A method comprising: detecting a page fault associated with a request for a device assigned to a guest operating system to perform direct memory access (DMA) of a requested page of memory; invalidating a mapping in a central processing unit (CPU) page table of a guest physical address to a host physical address for a candidate page for being swapped out of host memory; checking, by a hypervisor executing on a processing device, a DMA access state of the candidate page to determine whether or not the candidate page can be swapped out from the host memory; and in response to determining that the DMA access state for the candidate page indicates that the candidate page can be swapped out: removing the candidate page from the host memory; and invalidating a mapping in an input/output memory management unit (IOMMU) page table of the guest physical address to the host physical address for the candidate page.