1. A method of generating three-dimensional model information for a physical object for displaying said physical object as a three-dimensional virtual object according to a viewpoint position and an orientation of an observer, comprising: an acquisition step of acquiring a video image of physical space according to said viewpoint position and said orientation of said observer; an extraction step of extracting a region of said physical object from said video image of said physical space; a first generation step of generating a primitive virtual object having a position and orientation corresponding to a position and orientation of said physical object, said primitive virtual object having a size to encompass said extracted region; a second generation step of generating a virtual object corresponding to a shape of said extracted region from said primitive virtual object; and a model information generation step of generating three-dimensional model information for the virtual object generated in said second generation step as three-dimensional model information for said physical object.