Arduino as USB HID is slowed down when PC doesn't read events
For those that hit the same issue, here is how I solved it:

Expose the USB_SendSpace() information in the HID interface by editing

/usr/local/arduino-1.8.12/hardware/arduino/avr/libraries/HID/src/HID.cpp

u8 HID_::SendSpace(void)
{
        return USB_SendSpace(pluggedEndpoint);
}


And adding the interface to class HID_ in:

/usr/local/arduino-1.8.12/hardware/arduino/avr/libraries/HID/src/HID.h

That way, before calling HID().SendReport() you can check for send buffer space available by calling the HID().SendSpace() method.

If there is no space, just skip the SendReport() calls, and the Arduino loop() will no longer be slowed down by time outs.