1. A method for constructing a virtual model of an arbitrary, physical object based on video of the object acquired while the physical object is undergoing a deformation, the method comprising: identifying in the video a set of reference points corresponding to different positions on the physical object; for each reference point identified, observing a displacement of that reference point in response to the deformation of the physical object; grouping together a first group of reference points for which a shared first coordinate transform results in the observed displacement; grouping together a second group of reference points for which a shared second coordinate transform results in the observed displacement, the first and second coordinate transforms differing from each other; fitting the first group of reference points to a first 3D shape; fitting the second group of reference points to a second 3D shape; and combining at least the first and second 3D shapes to form the virtual model, the formed virtual model being of a prototype geometry not determined prior to the fitting of the first and second groups of reference points.