Does the PLL circuit which is used to increase frequency N times increase also the phase?
You have to be careful what you mean by phase, which always has to be measured with respect to something.

However, given that phase is the time integral of frequency, multiplying the frequency by N will also multiply the phase.

This means if there's a phase shift to the reference frequency, the output frequency will shift by N times that amount. If there's phase jitter on the input signal, there will be N times that amount on the output, if the jitter frequency is within the loop bandwidth.