The covariance of covariance (CoC) matrix is formed from the auto/cross-correlation function of acceleration responses of a structure under white noise ambient excitation. The components are function of the modal parameters of the structure and contain more information on the vibration modes of the structure compared to the general existing methods for extracting the modal parameters. This paper makes use of the CoC matrix and a new pattern match criterion for long-term health monitoring, damage localization and quantification of a five-bay three-dimensional frame structure. A large amount of measured data from an in-service suspension bridge is also analyzed. Only the acceleration responses are required to compute the covariance of covariance matrix. The components of the matrix are analyzed and the effects of the traffic flow and environmental temperature are studied. Finally, a strategy to identify the abnormal state of the bridge is presented based on the properties of the CoC components of the bridge. The CoC matrix is shown suitable for analyzing huge amount of measured data for the output-only structural damage detection without need of an analytical model.