How to find the time of beginning of the slope of the signal?
I observe the following about your signal:


of interest are only frequencies below 20 kHz
there's but a single, very dominant tone in that – you've got excellent SNR
The development of frequency over time is either constant, or a linear function of time


So, from that, I'd propose the following steps:


Low pass filter appropriately to restrict your bandwidth to below 15 to 20 kHz
employing a parametric spectrum estimator to find the frequency of your single tone
post-process that frequency estimate to decide whether you're still in the linear or in the constant frequency regime.


Regarding these steps:


might be easiest with just a simple second-order PLL. If you don't know where your tone is going to be at the start of your observation, start with a very high loop filter bandwidth, and as soon as you have a lock, reduce the bandwidth drastically. That second-order control loop actually directly gives you an error signal that you can use for 3.
that's a problem for a bit of low-pass filtering and estimating the frequency/time slope from that, and then checking whether the hypothesis still fits, or the slope has suddenly changed.