Although vision-based methods for displacement and vibration monitoring have been used in civil engineering for more than a decade, most of these techniques require physical targets attached to the structures. This requirement makes computer vision-based monitoring for real-life structures cumbersome due to need to access certain critical locations. In this study, a non-target computer vision-based method for displacement and vibration measurement is proposed by exploring a new type of virtual markers instead of physical targets. The key points of measurement positions obtained using a robust computer vision technique named scale-invariant feature transform show a potential ability to take the place of classical targets. To calculate the converting ratio between pixel-based displacement and engineering unit (millimetre), a practical camera calibration method is developed to convert pixel-based displacements to engineering unit since a calibration standard (a target) is not available. Methods and approaches to handle challenges such as low contrast, changing illumination and outliers in matching key points are also presented. The proposed method is verified and demonstrated on the UCF four-span bridge model and on a real-life structure, with excellent results for both static and dynamic behaviour of the two structures. Finally, the method requires a simple, less complicated and more cost-effective hardware compared to conventional displacement and vibration monitoring measuring technologies.