How to make bode plot when output signal changes amplitude? When I do frequency analysis on my feedback controlled system and the controller is really tightly tuned, I get a frequency response that looks like this:



Blue is excitation signal and green is system output signal. Does this mean that my system has nonlinear behavior? Or this just a condition that would be visible when hitting a zero or a pole? 

Smaller frequencies work fine and produce a sine wave output that is easy to interpret. It is higher frequencies that end up looking like this. 

Since I intend to use this to derive a bode plot, I'm also curious how do I interpret this data and what is the actual magnitude and phase on this plot for this frequency? It seems to me like it is fairly random. How would I go about sweeping frequencies and making a bode plot when many frequencies in this region look like this? FFT maybe on both signals? But how exactly?
You are probably still seeing other frequency components from the transient of the system. I assume you that you simulated your system for a fixed number of periods of the excitation signal. If this is the case then it would also explain why you only see the transient at high frequencies. This would be because a fixed number of high frequency periods take less time than the same number of low frequency periods. Therefore there is not enough time for the transient to die out for the high frequency excitation.