Frequency slows faster when it's decreasing
I think the issue is that you aren't ramping down the frequency from A to B, but the phase.  The phase needs to be integrated from the last previous value.
Plot the phase argument and you'll see what you did wrong.  The phase for source activity 1 should end up parallel but above the phase for source activity 2.
You say that the end frequency is 4 in both cases, but what you really have done is constrict both waveforms to have the same number of cycles.  Since source activity 1 starts faster, it must end slower.