How to do a factory reset when recovery mode, usb debugging and factory reset don't work?
The easiest way would be from fastboot by issuing a 'fastboot erase userdata' command. But booting afterwards might be troublesome because you will have no filesystem on your userdata partition! It is normal that you don't see anything on your screen except the LED when booting into fastboot mode using the key combo.

Just run fastboot as root on your computer (best on a live system with fastboot), then you don't need to mess with the device rules stuff. To me it seems very unlikely that the malware managed to manipulate your bootloader in a way that it still boots the system but does not allow entering fastboot mode.

Please be aware though that most malware cannot be removed by simply doing a factory reset as the factory reset only removes user data, not system stuff. If the malware managed to modify your boot or system partition (which is very likely) you will have to re-flash these partitions back to a stock state. Do you have access to an image of the stock firmware?