A damage detection methodology is proposed by integrating a nonlinear recursive filter and a non-contact computer vision based algorithm to measure structural dynamic responses. A phase-based optical flow algorithm inspired by the motion magnification technique is used to measure structural displacements, and the unscented Kalman filter is used to predict structural properties such as stiffness and damping coefficients. This non-contact displacement measurement methodology does not require an intensive instrumentation process, does not add any additional mass to the structure which may skew measurements, and can measure more signals compared to traditional methods. This measurement methodology still needs improvement as a tool due to its higher noise level relative to traditional accelerometer and laser vibrometer measurements. In order to detect structural damage using measured displacements from video, an unscented Kalman filter is used to remove noise from the displacement measurement and simultaneously detect damage by identifying the current stiffness and damping coefficient values, given a known mass, which are used to detect damage. To validate the proposed damage detection method state-space equations are derived without external excitation input and experimental tests are carried out. The experimental results show reasonable and accurate predictions of the stiffness and damping properties compared to dynamic analysis calculations. (C) 2016 Elsevier Ltd. All rights reserved.