Kalman Filtering with Unknown State Transition Matrix
Unfortunately, in Classical Kalman filter applications you need to have a guess of your state transition model governing the states. Classical choices for such models for time-varying systems are the random walk model, or its generalization, a first-order Markov model. If do not have a clue about which state model you can assume, you can try to extract a trend component from your state time series (via EMD or a moving average filter, for example), and plot it to have an idea of how the first-order moment of your states vary with time, for example. If it decays as a power law with $n$ (i.e. $E[x(n)] \approx \alpha^n$) until a final value, it may be an indication that a first-order Markov model with pole equal to $\alpha$ can serve as a state transition model.

Otherwise, you can treat the transition model for your states as a black box and consider techniques such as the Unscented Kalman Filter, for example.