USB direct connection and implemention on FPGA
It would be interesting to know if my hunch is correct or is it really
possible to do USB completely in hardware and how?

It's possible, but the FPGA would need differential transceivers that support 480MHz for USB 2.0, and I suppose that a clock running at 240MHz or more. Because this is difficult for many FPGA's most designs use UMTI which parallelizes the incoming serial stream.
For USB 1.2 I would think that cores are available and speed isn't that much of an issue since it runs at 12MHz
The figure below shows what is involved in implementing UMTI, if you could do this (I never have implemented on directly only a UMTI core) on an FPGA, then yes, you could run the data pins right into the FPGA. I am not aware of any cores that do this but that doesn't mean they aren't out there.

Source: https://www.intel.com/content/dam/www/public/us/en/documents/technical-specifications/usb2-transceiver-macrocell-interface-specifications.pdf
This is an open cores USB 2.0 diagram, the external phy implements the UMTI and the opencores usb 2.0 core processes the UMTI stream. In many cases it's better to have the trancievers outside of the FPGA or microprocessor even though it uses more GPIO.

Source: https://opencores.org/projects/usb
The orange crab uses a 12Mbit USB device which isn't hard to implement since it runs at 12MHz.
Full-speed (12Mbit) USB with a direct connection to the FPGA
It probably wouldn't be worth it to do USB on chip because it takes a lot of resources. If you spent 30$ or more for an FPGA and implementing USB uses 5 or 10% of resources, then it doesn't justify the cost of adding an external phy or USB to UART (which can be found for 1-2$). Unless you really need to save on space, I'd use an external phy or USB to Uart.