Is Virtual Memory unreliable?
At a hardware level, virtual memory means, essentially, a mapping from a virtual address space to a physical address space, the data structures and instructions to maintain them, and a facility to trap to the operating system if a mapping is not valid. This mapping is at the page level: a page maps to a page frame.

At the operating system level, we usually conceptualise a virtual address space as a collection of segments. A segment could represent part of a file which resides on secondary storage (e.g. disk). This could be program text, or a memory-mapped file. Or it could represent anonymous memory, that is, memory which is allocated by a process and may be swapped to secondary storage if not needed. Anonymous segments are often zero-fill, which isn't backed by physical memory until it is first used. Or it could represent a memory-mapped I/O area, or an operating system device, or an operating system data structure (e.g. the "u area" in Unix).

Segments can be shared between address spaces (e.g. after the Unix fork system call). They can be copy-on-write. And so on and so forth.

There are enough "things" that can be mapped which aren't really "memory" that you can't really say that virtual address space is only as useful as the amount of physical stuff that can be mapped to it.

One more thing: Another advantage of a large virtual address space is reduced external fragmentation. You see, the segments that comprise a virtual address space need to be allocated and deallocated just like anything else, and you can't really move them within the address space. So having a large virtual address makes it easier to allocate within it.