Do I have to debounce MCU reset pin?
It is not possible to physically damage an MCU from not debouncing the RESET input (assuming valid signal levels but invalid timing). It is possible to not reset it properly, which could lead to loss of data, EEPROM corruption, improper or undefined operation (heater stuck on, perhaps) and so on. 

A pushbutton switch is also a kind of open invitation for ESD to come into the system via the user's fat electrically charged finger (and similarly disposed body), and it is possible to fry the input or cause potentially destructive latch-up of the chip. Microchip (PIC) products seem to be particularly sensitive as per datasheet warnings, but a series resistor and shunt capacitor is virtually always a good idea. 

Since most reset inputs will have Schmitt trigger operation, this has the happy side-effect of guaranteeing proper minimum reset pulse width if you pick the part values sensibly. 

Added: Since you've mentioned it's a PIC18F, here are a couple notes from a typical PIC18F datasheet. 

Minimum reset (/MCLR) pulse width is 2usec: 



Warning about ESD on /MCLR input:



In general, I would recommend using an external supervisory chip to guarantee proper reset unless you can convince yourself that the built-in circuit is totally bulletproof.