Applying 5v to analogWrite Pin
Yes, depending on the current that will flow.

If your PWM output pin is connected to 5V without a current limiting resistor, nothing will happen during the HIGH phases of the PWM, but you will short the input pin during the LOW phases.

According to the manual of your motor controller, you should be able to use a current limiting resistor to prevent this.
Since the ATMega328P on your Arduino Nano can sink a maximum of 40 mA, that resistor should be at least 125 Ohms. A resistor larger than 1 kOhm should work fine.

The other option would be installing a diode between your Arduino Nano and the motor controller.
This does not require the use of a current limiting resistor.
The motor controller would still work, since the minimal voltage for the pin state to be detected as HIGH is 3V (1), much lower than the ~4.3V you're left with after a diode.