A structure under damage normally exhibits nonlinear dynamic behaviors and time-dependent stiffness and damping. By estimating instantaneous dynamic characteristics of a structures is generally very useful when assessing structural damage in real application. This study presents a novel and effective approach that accurately estimating instantaneous dynamic characteristic of a structure using time-varying autoregressive exogenous (TVARX) model on Gabor frame. To evaluate the time-varying coefficient matrices of a time-series model, the proposed procedure applies Gaussian-based weighted function and Fourier transform on time series model, then the Gabor-transform-based time series formula could be obtained. The dynamic characteristics of a structure are determined from the coefficient matrices. Numerical analyses demonstrate that the proposed approach is superior to traditional time-frequency analysis in accurately estimating instantaneous dynamic characteristics of a structure. Finally, the proposed approach is applied to process measured data for a steel frame specimen subjected to a series of base excitations in shaking table tests.