Frequency slows faster when it's decreasing I'm sure there's a simple explanation but I can't discover it. Consider two frequencies, a and b. a is higher than b. I can create a wave with both frequencies. But when I create a wave with frequency decreasing from a to b, the frequency decreases to even slower than frequency b. Consider the following MATLAB code.
timeVals = 0:0.001:1; 
sourceFreq = 8; 
freqPulse = linspace(1,.5,length(timeVals));
sourceActivity1 = cos(2 * pi * (sourceFreq .* freqPulse .* timeVals));

freqPulse = linspace(.5,.5,length(timeVals));
sourceActivity2 = cos(2 * pi * (sourceFreq .* freqPulse .* timeVals)); 

plot(sourceActivity1); hold on; plot(sourceActivity2)

Now, compare the two sourceActivities. The end frequency of sourceActivity1 is even slower than 4. And the frequency at the end of the wave is slower than sourceActivity2. This can be determined using visual inspection alone, or with spectrograms.
I think the issue is that you aren't ramping down the frequency from A to B, but the phase.  The phase needs to be integrated from the last previous value.
Plot the phase argument and you'll see what you did wrong.  The phase for source activity 1 should end up parallel but above the phase for source activity 2.
You say that the end frequency is 4 in both cases, but what you really have done is constrict both waveforms to have the same number of cycles.  Since source activity 1 starts faster, it must end slower.