PIC does not restart in case of power off and then power on
but it looks like it continue from where it was when I close the power source. Is it normal?


No, that's not normal (assuming that your code isn't written in a way that it logs the current function into flash and deliberately jumps to there on power-on, or similar).

There is a common cause for your described behaviour. This can be caused by an external voltage continuing to be applied to the MCU, even after its main power (Vdd) is removed. The external voltage then continues to power the MCU through its ESD protection diodes (or equivalent structures).

In some cases, enough power might be supplied for the MCU to continue running (apparently normally). In other cases, there is just enough power for the MCU not to "lose state" when main power is removed, so that it appears to continue where it last stopped, when main power is next reapplied.

Make a reproducible test case using your hardware, where removing and reapplying main power to the MCU definitely does not cause the expected full restart. Then after you are sure that you have "the problem configuration", remove main power and measure all the voltages on all the MCU pins, relative to the MCU Gnd (Vss). If this hypothesis is correct, you will find one or more pins with a non-zero voltage being applied to the MCU.

If you add into the question a full schematic and explanation of all voltage sources, that will allow readers to review it and help confirm or deny possible problems in this area.