Is everything on a computer represented simply by just "voltages"?
It is correct for all the volatile state of the machine, i.e. the processor state, the RAM.

For the non-volatile part, it is something else that, when the machine is powered on, can be converted into voltage levels representing 0 or 1.

The "something else" is in general one of two things: magnetic polarization state (for the HD, and the old-fashioned floppy disks) or electrostactic state (for EEPROM, Flash &amp; etc.). But may also be optical properties, like in a CDROM.

Although Flash memory &amp; EEPROM use floating gate transistors where, maybe, you could also say the trapped electrons in the gate are a "voltage" (a DP from the rest of the circuit).