TECHNICAL FIELD 
       [0001]    The present disclosure relates to image processing apparatus and method, and a program, in particular, image processing apparatus and method, and a program, which make it possible to obtain stereoscopic images of a subject seen from a plurality of different viewpoint positions more easily. 
       BACKGROUND ART 
       [0002]    In the related art, a technique is known which displays a subject stereoscopically by using a pair of images obtained by capturing the same subject from different viewpoints (see, for example, PTL 1). 
         [0003]    For example, in the case of a typical camera, a subject is captured in a state in which the optical axis of the camera is substantially horizontal to the ground, and the obtained image is projected onto a screen that is perpendicular to the ground. In this case, it can be supposed that the viewpoint position of the user who sees the displayed image is located on a straight line that is substantially perpendicular to the screen on which the image is displayed. 
         [0004]    Consequently, when an image is to be displayed stereoscopically with such a display system, the binocular disparity/parallax of the user is expressed as a shift of the image to the left and right. That is, the subject is displayed stereoscopically by use of a pair of images having a disparity in a direction which is perpendicular to the user&#39;s sight line direction and along which the user&#39;s left and right eyes are arranged. 
       CITATION LIST 
     Patent Literature 
       [0005]    PTL 1: Japanese Unexamined Patent Application Publication No. 2010-8575 
       SUMMARY OF INVENTION 
     Technical Problem 
       [0006]    On the other hand, as shown in  FIG. 1 , in the case of a display system with which a subject is captured by a camera  11  in a state in which the optical axis of the camera  11  lies in a direction perpendicular to the ground, and the obtained image is projected onto a screen parallel to the ground, the angle of rotation with respect to the direction of the vertical axis of the screen that is parallel to the ground remains as a degree of freedom. It should be noted that in  FIG. 1 , the right diagonal direction, the left diagonal direction, and the vertical direction that are orthogonal to each other indicate the x-direction, the y-direction, and the z-direction, respectively. 
         [0007]    In the example of  FIG. 1 , a subject H 11  is captured by the camera  11  in a state in which the optical axis of the camera  11  is parallel to the z-direction, that is, from above in the drawing. The image of the subject H 11  is displayed on a screen that is parallel to the xy-plane. 
         [0008]    Consequently, such a display system allows the user to look into the screen from various directions with the z-axis as the axis of rotation, making it difficult to display the subject H 11  stereoscopically. 
         [0009]    In the case where an image is displayed using a naked eye display scheme represented by the parallax barrier scheme or the lenticular scheme as the method of displaying a stereoscopic image, the direction of disparity is physically restricted by the direction of arrangement of slits or lenticular lenses provided in a display device for the image. For example, if slits or lenticular lenses are arranged in the x-direction in the drawing, the subject H 11  is displayed stereoscopically by use of an image pair having a disparity in the x-direction. Consequently, the subject H 11  is not observed in a stereoscopic manner unless the user sees the screen in such a way that the sight line direction becomes parallel to the y-direction. 
         [0010]    It should be noted that in the case where an image is displayed stereoscopically by the liquid crystal shutter scheme (time division shutter scheme), unlike in the case of the naked eye display scheme, there is no physical restriction on the direction of disparity. Thus, in principle, it is possible to display the subject H 11  stereoscopically no matter from which direction the user sees the screen. 
         [0011]    However, in the case of displaying a stereoscopic image by the liquid crystal shutter scheme, an image pair having a disparity in a direction perpendicular to the user&#39;s sight line direction is required. For example, if the user&#39;s sight line direction is a direction parallel to the x-direction, an image pair having a disparity in the y-direction is required for stereoscopic display of an image, and if the user&#39;s sight line direction is a direction parallel to the y-direction, an image pair having a disparity in the x-direction is required. 
         [0012]    In this way, in the liquid crystal shutter scheme, if an image pair having a disparity in a direction perpendicular to the user&#39;s sight line direction is used, it is possible to display the subject H 11  stereoscopically irrespective of the user&#39;s sight line direction. However, usually, a change in the user&#39;s sight line direction is not assumed for images such as contents intended for stereoscopic vision. Accordingly, only an image pair having a disparity in a specific direction is prepared, and thus it has been impossible to deal with a change in the user&#39;s sight line direction. 
         [0013]    The disclosed embodiments of the present invention have been made in view of the above circumstances, and make it possible to obtain stereoscopic images of a subject seen from a plurality of different viewpoint positions more easily. 
       Solution to Problem 
       [0014]    There is disclosed an apparatus for processing source images having known viewpoints. The apparatus may include a viewpoint determining section for determining, relative to the known viewpoints, left-eye and right-eye viewpoints of a viewer. Additionally, the apparatus may include a disparity image generating section. The disparity generating section may be for selecting data of at least one of the source images, based on the determined left-eye viewpoint, as raw left-eye data. In addition, the disparity generating section may be for selecting data of at least one of the source images, based on the determined right-eye viewpoint, as raw right-eye data. The disparity generating section may also be for outputting left-eye image data, based on the raw left-eye data. Additionally, the disparity generating section may be for outputting right-eye image data, based on the raw right-eye data. 
         [0015]    There is also disclosed a method of processing source images having known viewpoints. A processor may execute a program to cause an apparatus to perform the method. The program may be stored on a computer-readable medium. The method may include determining, relative to the known viewpoints, left-eye and right-eye viewpoints of a viewer. The method may also include selecting data of at least one of the source images, based on the determined left eye viewpoint, as raw left-eye data. In addition, the method may include selecting data of at least one of the source images, based on the determined right-eye viewpoint, as raw right-eye data. The method may also include outputting left-eye image data, based on the raw left-eye data. Additionally, the method may include outputting right-eye image data, based on the raw right-eye data. 
       Advantageous Effects of Invention 
       [0016]    According to an aspect of the disclosed embodiments, it is possible to obtain stereoscopic images of a subject seen from a plurality of different viewpoint positions more easily. 
     
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         [0017]      FIG. 1  is a diagram for explaining the relationship between an imaging system and a display system for a stereoscopic image according to the related art. 
           [0018]      FIG. 2  is a diagram explaining about display of a stereoscopic image by an image processing apparatus consistent with an embodiment of the present invention. 
           [0019]      FIG. 3  is a diagram explaining about source images. 
           [0020]      FIG. 4  is a diagram showing an example of the configuration of an image processing apparatus consistent with an embodiment of the present invention. 
           [0021]      FIG. 5  is a flowchart explaining about a stereoscopic display process. 
           [0022]      FIG. 6  is a diagram explaining about display viewpoints of disparity images. 
           [0023]      FIG. 7  is a diagram explaining about generation of a disparity image. 
           [0024]      FIG. 8  is a diagram showing another example of the configuration of the image processing apparatus. 
           [0025]      FIG. 9  is a flowchart explaining about a stereoscopic display process. 
           [0026]      FIG. 10  is a diagram explaining about generation of recording source images. 
           [0027]      FIG. 11  is a diagram explaining about generation of a disparity image. 
           [0028]      FIG. 12  is a flowchart explaining about a stereoscopic display process. 
           [0029]      FIG. 13  is a diagram explaining about generation of a disparity image. 
           [0030]      FIG. 14  is a diagram showing another example of the configuration of the image processing apparatus. 
           [0031]      FIG. 15  is a flowchart explaining about a stereoscopic display process. 
           [0032]      FIG. 16  is a diagram showing another example of the configuration of the image processing apparatus. 
           [0033]      FIG. 17  is a flowchart explaining about a stereoscopic display process. 
           [0034]      FIG. 18  is a flowchart explaining about a stereoscopic display process. 
           [0035]      FIG. 19  is a block diagram showing an example of the configuration of a computer. 
       
    
    
     DESCRIPTION OF EMBODIMENTS 
       [0036]    Hereinbelow, embodiments of the present invention will be described with reference to the drawings. 
       First Embodiment 
       [0037]    Overview of Present Invention 
         [0038]    For example, as shown in  FIG. 2 , an image processing apparatus consistent with an embodiment of the invention displays a subject in a stereoscopic manner on a display section  41  whose display screen is horizontal to the ground. It should be noted that in  FIG. 2 , the vertical direction, the lateral direction, and the depth direction respectively indicate the y-direction, the x-direction, and the z-direction. 
         [0039]    The image processing apparatus displays a stereoscopic image by alternately displaying disparity images having a disparity with respect to each other, on a screen parallel to the xy-plane by the liquid crystal shutter scheme, for example. That is, the subject on the disparity images is displayed stereoscopically. At this time, the image processing apparatus detects the position on the xy-plane of a user who is seeing the displayed stereoscopic image, and determines the user&#39;s sight line direction from the position. 
         [0040]    It should be noted that the user is supposed to view the stereoscopic image so as to look down the display section  41  substantially in the z-direction, from the vicinity of the edge of the display section  41 . Also, the user&#39;s sight line direction is supposed to be a direction parallel to the xy-plane. 
         [0041]    Upon determining the user&#39;s sight line direction, on the basis of the sight line direction, the image processing apparatus controls display of the stereoscopic image so that when the stereoscopic image is seen from the viewpoint position of the user, the subject on the stereoscopic image is observed in a stereoscopic manner. 
         [0042]    More specifically, as shown in the upper left of  FIG. 3 , the image processing apparatus records a plurality of images (i.e., data of a plurality of images) (hereinafter, referred to as source images) having a disparity with respect to each other, which are obtained by capturing a specific object at each point in the xy-plane taken as a viewpoint. 
         [0043]    It should be noted that in  FIG. 3 , the vertical direction and the lateral direction indicate the y-direction and the x-direction, respectively. Also, in the drawing, each single rectangle represents a single source image, and in the drawing, each single circle in the lower right represents the viewpoint of a single source image. 
         [0044]    In the example of  FIG. 3 , a total of 25, 5 (x-direction)*5 (y-direction) source images are prepared in advance. In  FIG. 3 , the positional relationship between the source images arranged in an array is the same as the positional relationship between the viewpoint positions of the respective source images. Therefore, these source images are images in which the same object seen from mutually different viewpoints is displayed. 
         [0045]    In the following, the source images arranged in an array as shown in the upper left of  FIG. 3  are considered as a set of grid points on the xy-plane by taking the viewpoint position of each of the source images as a grid point, as shown in the lower right in the drawing. Here, in the drawing, in the vicinity of each of the viewpoints in the lower right, coordinates of the viewpoint in the xy-coordinate system are shown. For example, in the drawing, of the set of grid points in the lower right, ( 1 ,  1 ) located in the vicinity of the viewpoint at the left edge of the uppermost stage represents the coordinates (x, y) in the xy-coordinate system of the viewpoint. That is, the coordinates ( 1 ,  1 ) represent the position of the viewpoint whose x-coordinate and y-coordinate are both  1 . 
         [0046]    In the image processing apparatus, disparity images are generated from the source images of individual viewpoints, with the set of grid points on the xy-plane taken as being located on the foreground side of the display section  41  in  FIG. 2 . That is, on the xyz-space, the positional relationship between the center of the display screen of the display section  41 , and the viewpoint of each source image is supposed to be the same as the positional relationship between the subject of the source images and the viewpoint of each source image. 
         [0047]    It should be noted that such source images are captured by a light-field camera or the like with which multi-viewpoint images can be acquired simultaneously while being arranged in an array. Also, the source images may be still images or may be moving images. 
         [0048]    From the user&#39;s sight line direction and a disparity d of the disparity images to be displayed, the image processing apparatus determines a pair of points (hereinafter, referred to as display viewpoints) on the xy-plane which are arranged in a direction perpendicular to the sight line direction and whose distance is the same length as the disparity d. Then, with respect to each of the display viewpoints, the image processing apparatus selects several source images whose viewpoint positions (i.e., known viewpoints) are located in the vicinity of the display viewpoint, generates the disparity images of the respective display viewpoints from the selected source images, and displays a stereoscopic image on the basis of the obtained disparity image pair. 
         [0049]    Since the pair of disparity images obtained in this way has the disparity d in a direction perpendicular to the user&#39;s sight line direction on the xy-plane, the subject on the disparity images is observed in a stereoscopic manner by the user. 
         [0050]    &lt;Configuration of Image Processing Apparatus&gt; 
         [0051]    Next, a specific embodiment of the image processing apparatus described above will be described.  FIG. 4  is a diagram showing an example of the configuration of an image processing apparatus consistent with an embodiment of the present invention. It should be noted that in the drawing, portions corresponding to those in the case of  FIG. 2  are denoted by the same symbols, and description thereof is omitted as appropriate. 
         [0052]    An image processing apparatus  71  in  FIG. 4  includes the display section  41 , an acquisition section  81 , a recording section  82 , a sensor  83 , an image processing section  84 , and a display control section  85 . 
         [0053]    The display section  41  is set so that the display screen becomes substantially perpendicular to the vertical direction, that is, parallel to the ground. The display section  41  displays a stereoscopic image by the liquid crystal shutter scheme on the basis of disparity images supplied from the display control section  85 . 
         [0054]    The acquisition section  81  acquires the source images described with reference to  FIG. 3  from, for example, a removable medium or a camera mounted on the image processing apparatus  71 , and supplies the source images to the recording section  82  for recording. The recording section  82  records the source images supplied from the acquisition section  81 , and supplies the source images to the image processing section  84  as required. More specifically, the recording section  82  records the individual source images, and the viewpoint positions of those source images in association with each other. 
         [0055]    The sensor  83  is formed by a plurality of human sensors, image sensors, or the like, which are provided to a table supporting the display section  41  or the housing of the display section  41  so as to surround the display screen of the display section  41 . The sensor  83  detects the position of a user (human) who is present in the vicinity of the display section  41 , and supplies the detection result to the image processing section  84 . 
         [0056]    The image processing section  84  generates a pair of disparity images from the detection result supplied from the sensor  83 , and the source images recorded in the recording section  82 . The image processing section  84  includes a sight line direction determining section  101 , a viewpoint position determining section (i.e., a viewpoint determining section)  102 , and a disparity image generating section  103 . 
         [0057]    The sight line direction determining section  101  determines the user&#39;s sight line direction from the detection result of the position of the user supplied from the sensor  83 , and supplies the obtained sight line direction to the viewpoint position determining section  102 . The viewpoint position determining section  102  determines display viewpoints that are the viewpoint positions of disparity images, from the disparity d that is specified in advance, and the sight line direction from the sight line direction determining section  101 , and supplies the display viewpoints to the disparity image generating section  103 . 
         [0058]    The disparity image generating section  103  reads necessary source images from the recording section  82  on the basis of the display viewpoints supplied from the viewpoint position determining section  102 , and generates disparity images on the basis of the read source images. The disparity image generating section  103  supplies the pair of generated disparity images to the display control section  85 . The display control section  85  supplies the pair of disparity images supplied from the disparity image generating section  103  to the display section  41 , and causes the disparity images to be displayed stereoscopically. 
         [0059]    &lt;Description of Stereoscopic Display Process&gt; 
         [0060]    Incidentally, when the user operates the image processing apparatus  71  to instruct display of a stereoscopic image, the image processing apparatus  71  performs a stereoscopic display process in response to the user&#39;s instruction, and causes a stereoscopic image to be displayed. Hereinbelow, referring to the flowchart in  FIG. 5 , the stereoscopic display process by the image processing apparatus  71  will be described. 
         [0061]    In step S 11 , the sight line direction determining section  101  determines the user&#39;s sight line direction. That is, the sensor  83  detects the position of a user who is present around the display section  41 , and supplies the detection result to the sight line direction determining section  101 . Then, the sight line direction determining section  101  determines the user&#39;s sight line direction from the detection result from the sensor  83 , and supplies the sight line direction to the viewpoint position determining section  102 . For example, the direction of a vector whose starting point is the position of the user on a plane parallel to the xy-plane, and whose end point is the position of the center of the display screen of the display section  41  is defined as the sight line direction. 
         [0062]    It should be noted that in the case when a plurality of users are detected around the display section  41  by the sensor  83 , one of the users is selected, and the sight line direction of the user is determined. In the image processing apparatus  71 , while the direction of disparity of disparity images also changes when the user&#39;s sight line direction changes, for the sake of the temporal stability of the disparity images, it is desirable that once a single user is selected from among a plurality of users, the user be selected continuously during playback of a stereoscopic image. 
         [0063]    In step S 12 , the viewpoint position determining section  102  finds a pair of display viewpoints (i.e., left-eye and right-eye viewpoints) of disparity images from the user&#39;s sight line direction supplied from the sight line direction determining section  101 , and the disparity d specified in advance. Here, the disparity d is, for example, the length of separation of the user&#39;s eyes assumed at the time of viewing of a stereoscopic image, which may be specified by the user or may be defined in advance. 
         [0064]    For example, as shown in  FIG. 6 , the viewpoint position determining section  102  virtually draws a circle SC 11  with a radius d/2, which is centered at the center position (hereinafter, referred to as reference position) of a set of individual viewpoints (grid points) of source images, on the xy-plane. It should be noted that in  FIG. 6 , the vertical direction and the lateral direction respectively indicate the y-direction and the x-direction, and each single circle represents the viewpoint of a single source image. 
         [0065]    Upon drawing the circle SC 11  with the radius d/2, the viewpoint position determining section  102  then finds the angle theta formed by a user&#39;s sight line direction SD 11 , and a virtual x-axis AX 11  parallel to the x-direction, and finds a tangent L 11  and a tangent L 12  to the circle SC 11  at the angle theta. These tangent L 11  and tangent L 12  are each a straight line that is parallel to the sight line direction SD 11 . 
         [0066]    Upon finding the tangent L 11  and the tangent L 12  in this way, the viewpoint position determining section  102  defines the tangent points between these obtained tangent L 11  and tangent L 12 , and the circle SC 11  as a pair of display viewpoints. In the example of  FIG. 6 , a display viewpoint HS 11  and a display viewpoint HD 12  are obtained, the distance between these display viewpoint HS 11  and display viewpoint HS 12  is the disparity d between the disparity images to be displayed, and the straight line connecting the display viewpoint HS 11  and the display viewpoint HS 12  is orthogonal to the sight line direction SD 11 . Therefore, if a pair of disparity images whose viewpoint positions are these display viewpoint HS 11  and display viewpoint HS 12  is displayed stereoscopically, then the user who sees the disparity images from the sight line direction SD 11  should be able to see the subject on the disparity images in a stereoscopic manner. 
         [0067]    It should be noted that the display viewpoint HS 11  is the viewpoint of a disparity image for the left eye which is observed by the left eye of the user, and the display viewpoint HS 12  is the viewpoint of a disparity image for the right eye which is observed by the right eye of the user. As these disparity images for the right eye and the left eye are displayed by the liquid crystal shutter scheme, a stereoscopic image is displayed. 
         [0068]    Returning to the description of the flowchart in  FIG. 5 , after finding the pair of display viewpoints, the viewpoint position determining section  102  supplies the found display viewpoints to the disparity image generating section  103 , and the processing proceeds to step S 13 . 
         [0069]    In step S 13 , on the basis of the display viewpoints supplied from the viewpoint position determining section  102 , the disparity image generating section  103  selects several source images whose viewpoints are located in the vicinity of each of the display viewpoint on the xy-plane. 
         [0070]    For example, in the example of  FIG. 6 , the disparity image generating section  103  selects four viewpoints SP 11  to SP 14  located in the vicinity of the display viewpoint HS 11  and surrounding the display viewpoint HS 11  (i.e., four viewpoints positioned at the corners of a rectangle), selects the source images of those viewpoints (i.e., raw left-eye data), and also reads the selected source images from the recording section  82 . Likewise, the disparity image generating section  103  selects four viewpoints whose viewpoints are located in the vicinity of the display viewpoint HS 12  (i.e., four viewpoints positioned at the corners of a rectangle), selects the source images of those viewpoints (i.e., raw right-eye data), and also reads the selected source images from the recording section  82 . 
         [0071]    In step S 14 , for each display viewpoint, the disparity image generating section  103  generates a disparity image (i.e., finished image data including at least one of left-eye or right-eye image data) by using the source images of the viewpoints located in the vicinity of the display viewpoint, which are selected in step S 13 . 
         [0072]    For example, as shown in  FIG. 7 , the disparity image generating section  103  generates a disparity image whose viewpoint is the display viewpoint HS 11 , by synthesizing the source images whose viewpoints are located in the vicinity of the display viewpoint HS 11 . It should be noted that  FIG. 7  is a diagram showing the vicinity of the display viewpoint HS 11  in  FIG. 6  in an enlarged fashion. In the drawing, portions corresponding to those in the case of  FIG. 6  are denoted by the same symbols, and description thereof is omitted. Also, in  FIG. 7 , the vertical direction and the lateral direction respectively indicate the y-direction and the x-direction. 
         [0073]    First, the disparity image generating section  103  synthesizes the source image of the viewpoint SP 11  located at coordinates (x′, y′), and the source image of the viewpoint SP 12  located at coordinates (x′, y′+1) by a morphing method or the like, thereby generating the image of a viewpoint Gil located at coordinates (x′, y). 
         [0074]    For example, in the synthesis of the source images, the pixel values of pixels in the image of the viewpoint G 11  can be found by interpolating the pixel values of pixels in the two source images at a ratio according to the distances from the viewpoint G 11  to the viewpoint SP 11  and the viewpoint SP 12 . 
         [0075]    After generating the image of the viewpoint Gil in this way, the disparity image generating section  103  likewise synthesizes the source image of the viewpoint SP 13  located at coordinates (x′+1, y′), and the source image of the viewpoint SP 14  located at coordinates (x′+1, y′+1) by a morphing method or the like, thereby generating the image of a viewpoint G 12  located at coordinates (x′+1, y). 
         [0076]    Further, the disparity image generating section  103  synthesizes the image of the viewpoint Gil and the image of the viewpoint G 12  by a morphing method or the like, thereby generating the disparity image of the display viewpoint HS 11  located at coordinates (x, y). It should be noted that in the example of  FIG. 7 , after synthesis of source images arranged in the y-direction is performed, synthesis of images arranged in the x-direction is performed to thereby generate a disparity image. However, it is also possible to perform synthesis of source images arranged in the x-direction, and then perform synthesis of images arranged in the y-direction to thereby generate a disparity image. 
         [0077]    Returning to the description of the flowchart in  FIG. 5 , after generating the disparity images of the pair of display viewpoints, the disparity image generating section  103  supplies the generated disparity image pair to the display control section  85 , and the processing proceeds to step S 15 . 
         [0078]    In step S 15 , the display control section  85  supplies the disparity image pair supplied from the disparity image generating section  103  to the display section  41 , and causes the disparity image pair to be displayed stereoscopically by the liquid crystal shutter scheme. Thus, the disparity images can be seen in a stereoscopic manner by the user who is seeing the display section  41  while wearing dedicated glasses provided with liquid crystal shutters. That is, a stereoscopic image is observed. 
         [0079]    In step S 16 , the sight line direction determining section  101  judges whether or not the user&#39;s sight line direction has changed. For example, the sight line direction is judged to have changed when a change in the user&#39;s sight line direction is detected from the detection result on the user supplied from the sensor  83 . 
         [0080]    If it is judged in step S 16  that the sight line direction has changed, the processing returns to step S 11 , and the above-described process is repeated. That is, the user&#39;s new sight line direction is detected, a disparity image pair having a disparity in a direction perpendicular to the sight line direction is generated, and a stereoscopic image is displayed. 
         [0081]    On the other hand, if it is judged in step S 16  that the sight line direction has not changed, in step S 17 , the image processing apparatus  71  judges whether or not to end the process of displaying a stereoscopic image. For example, it is judged to end the processing when the image processing apparatus  71  is operated by the user, and it is instructed to end display of a stereoscopic image. 
         [0082]    If it is judged in step S 17  not to end the processing, the processing returns to step S 16 , and the above-described process is repeated. That is, the same stereoscopic image is displayed continuously until the user&#39;s sight line direction changes. 
         [0083]    On the other hand, if it is judged in step S 17  to end the processing, the image processing apparatus  71  causes display of a stereoscopic image to end, and the stereoscopic display process ends. 
         [0084]    In the above-mentioned manner, the image processing apparatus  71  detects the sight line direction of a user who is present around the display section  41 , and finds the display viewpoint of each disparity image to be displayed from the sight line direction. Then, the image processing apparatus  71  synthesizes, among source images of a plurality of different viewpoints, source images whose viewpoints are located in the vicinity of the display viewpoint, thereby generating the disparity image of the display viewpoint. 
         [0085]    In this way, by detecting the user&#39;s sight line direction, finding display viewpoints of disparity images from the sight line direction, and generating the disparity images from source images whose viewpoints are located in the vicinity of their display viewpoints, it is possible to easily obtain disparity images with which a subject is observed in a stereoscopic manner by the user, irrespective of the viewpoint position of the user. 
         [0086]    Moreover, according to the image processing apparatus  71 , disparity images corresponding to an arbitrary sight line direction can be easily obtained. Thus, even when the user&#39;s sight line direction changes during viewing of a stereoscopic image, following the change in sight line direction, disparity images having a disparity in a different direction can be displayed stereoscopically. That is, no matter from which position  360  degrees around the display section  41  the user sees the display screen of the display section  41 , the subject on the disparity images can be seen in a stereoscopic manner by the user. 
         [0087]    As described above, according to the image processing apparatus  71 , by using existing three-dimensional image display devices, a new capability is realized, whereby a stereoscopic image is displayed in such a way that a subject can be viewed stereoscopically from a free viewpoint of 360 degrees. It should be noted that the image processing apparatus  71  as described above can be applied to information displays of a type embedded in a table, projectors for surgical images, and the like. 
         [0088]    It is particularly effective to apply the image processing apparatus  71  to, for example, playback systems which capture the operating part being surgically operated on by a light-field camera from directly above, and reconstruct the operating part seen from various positions in a three-dimensional fashion for stereoscopic display. In such a case, source images captured by the light-field camera are supplied to the image processing apparatus  71 . In the image processing apparatus  71 , from these source images, images during the surgical operation are generated as disparity images, and displayed stereoscopically. 
         [0089]    In playback systems according to the related art, only disparity images on a fixed baseline are displayed stereoscopically. However, according to the image processing apparatus  71 , a subject can be displayed stereoscopically from whatever direction, without being restricted to a specific baseline. Thus, it is possible to recreate movements of instruments during the operation, and the shape of the organ or the like to be surgically operated on, with a greater sense of realism. Also, it is possible to view images during the operation from the viewpoint of the surgical operator. 
         [0090]    It should be noted that the foregoing description is directed to the case in which the disparity d is not larger than the maximum separation between viewpoints in a set of viewpoints (grid points) of source images. In this case, each disparity image is generated through synthesis by interpolation of source images whose viewpoints are located in the vicinity of its display viewpoint. 
         [0091]    However, without limitation to this case, the display d may be larger than the maximum separation between viewpoints in a set of viewpoints of source images. In such a case, each disparity image is generated through synthesis by interpolation and extrapolation of source images of the four nearest viewpoints surrounding its display viewpoint. 
         [0092]    Also, in the case where the separation of viewpoints of source images is sufficiently narrow, and spatial resolution is excellent, instead of generating the disparity image of each display viewpoint, a source image whose viewpoint is located closest to the display viewpoint may be displayed as a disparity image as it is. In this case as well, it is possible to obtain substantially the same effect as that in the case of generating each disparity image from several source images, thus allowing a subject to be displayed in a sufficiently stereoscopic manner. 
       Second Embodiment &lt;Configuration of Image Processing Apparatus&gt; 
       [0093]    Also, in the above-mentioned case, supposing that the disparity d that is the separation of the user&#39;s eyes is constant, it follows that display viewpoints are located on a circumference with a radius d/2, irrespective of the user&#39;s sight line direction. Accordingly, to compress the amount of information of the group of source images to be recorded in advance, by using acquired source images, source images whose viewpoints are located on the circumference may be generated for the purpose of recording, and only the generated source images whose viewpoints are located on the circumference (hereinafter, also referred to as recording source images) may be recorded in advance. 
         [0094]    In such a case, the image processing apparatus  71  is configured as shown in  FIG. 8 , for example. 
         [0095]    In the image processing apparatus  71  shown in  FIG. 8 , a recording source image generating section  131  is further provided in addition to the image processing apparatus  71  in  FIG. 4 . Otherwise, the configuration is the same as that of the image processing apparatus  71  in  FIG. 4 . 
         [0096]    That is, in the image processing apparatus  71  in  FIG. 8 , the recording source image generating section  131  is provided between the acquisition section  81  and the recording section  82 . It should be noted that in  FIG. 8 , portions corresponding to those in the case of  FIG. 4  are denoted by the same symbols, and description thereof is omitted as appropriate. 
         [0097]    The recording source image generating section  131  generates recording source images whose viewpoints are located at predetermined positions on the xy-plane, from source images acquired by the acquisition section  81 , and supplies the recording source images to the recording section  82 . It should be noted that the generated recording source images are recoded to the recording section  82 . 
         [0098]    &lt;Description of Stereoscopic Display Process&gt; 
         [0099]    Next, referring to the flowchart in  FIG. 9 , a stereoscopic display process performed by the image processing apparatus  71  in  FIG. 8  will be described. 
         [0100]    In step S 41 , the recording source image generating section  131  defines viewpoint positions of recording source images on the basis of the disparity d that is specified in advance. 
         [0101]    Specifically, the acquisition section  81  acquires a plurality of source images and supplies the source images to the recording source image generating section  131 . Then, for example, as shown in  FIG. 10 , the recording source image generating section  131  virtually draws the circle SC 11  with a radius d/2, which is centered at the reference position of a set of viewpoints (grid points) of the source images, on the xy-plane. 
         [0102]    It should be noted that in  FIG. 10 , the vertical direction and the lateral direction respectively indicate the y-direction and the x-direction, and each single circle located at a position different from a position on the circumference of the circle SC 11  represents the viewpoint of a single source image. Also, in  FIG. 10 , portions corresponding to those in the case of  FIG. 6  are denoted by the same symbols, and description thereof is omitted. 
         [0103]    Upon drawing the circle SC 11  on the xy-plane, the recording source image generating section  131  then defines  12  points that divide the circumference of the circle SC 11  into equal parts, as viewpoints VP 1  to VP 12  of recording source images. It should be noted that the number and positions of points (viewpoints) that divide the circle SC 11  into equal parts are arbitrary number and positions. Also, the viewpoints of recording source images may not necessarily be points that divide the circumference of the circle SC 11  into equal parts. 
         [0104]    In step S 42 , the recording source image generating section  131  generates the recording source images of the respective viewpoints by using the source images supplied from the acquisition section  81 , and supplies the recording source images to the recording section  82 . 
         [0105]    For example, the recording source image generating section  131  generates a total of  12  recording source images whose respective viewpoints are the viewpoints VP 1  to VP 12  in  FIG. 10 . Specifically, for the viewpoint VP 1 , the viewpoint VP 4 , the viewpoint VP 7 , and the viewpoint VP 10 , source images whose viewpoints are located at the same positions already exist, and thus those source images are defined as the respective recording source images of the viewpoint VP 1 , the viewpoint VP 4 , the viewpoint VP 7 , and the viewpoint VP 10 . 
         [0106]    Also, each of the recording source images of the other viewpoints is generated through synthesis by the morphing method of the four nearest viewpoints surrounding the viewpoint of the recording source image. That is, in the example of  FIG. 10 , each of the recording source images of the viewpoint VP 2 , the viewpoint VP 3 , the viewpoint VP 5 , the viewpoint VP 6 , the viewpoint VP 8 , the viewpoint VP 9 , the viewpoint VP 11 , and the viewpoint VP 12  is generated through synthesis by interpolation of source images whose viewpoints are located in the vicinity. 
         [0107]    After the recording source images are obtained in this way, the recording source image generating section  131  supplies the generated recording source images to the recording section  82 . Then, in step S 43 , the recording section  82  records the recording source images supplied from the recording source image generating section  131 . 
         [0108]    In this way, by generating recording source images whose viewpoints are located on the circumference with a radius d/2 which is centered at the reference position on the xy-plane, and recording the recording source images in advance, the amount of images to be recorded in advance can be reduced. For example, in the example of  FIG. 10 , only  12  recording source images need to be recorded in advance, when it is otherwise necessary to record  25  source images in advance. 
         [0109]    Once the recording source images are recorded, thereafter, the processes in step S 44  and step S 45  are performed, and display viewpoints are found. Since these processes are the same as the processes in step S 11  and step S 12  in  FIG. 5 , description thereof is omitted. 
         [0110]    In step S 46 , for each of the display viewpoints supplied from the viewpoint position determining section  102 , the disparity image generating section  103  selects recording source images whose viewpoints are located in the vicinity of the display viewpoint (i.e., raw left-eye data or raw right-eye data). 
         [0111]    Specifically, since a point located on the circle SC 11  in  FIG. 10  is found as each display viewpoint, the recording source images of viewpoints that are adjacent to and on both sides of the display viewpoint on the circle SC 11  among the viewpoints VP 1  to VP 12  are selected. For example, if one display viewpoint is located between the viewpoint VP 1  and the viewpoint VP 2  on the circle SC 11 , the recording source image of the viewpoint VP 1 , and the recording source image of the viewpoint VP 2  are selected. 
         [0112]    Upon selecting recording source images whose viewpoints are located in the vicinity of the display viewpoint, the disparity image generating section  103  reads the selected recording source images from the recording section  82 , and the processing proceeds to step S 47 . 
         [0113]    In step S 47 , the disparity image generating section  103  finds the intersection point between a straight line connecting the display viewpoint and the farther viewpoint of the viewpoints of the selected recording source images from the display viewpoint, and a straight line connecting the nearer viewpoint of the viewpoints of the selected recording source images to the display viewpoint and the reference position. This intersection point (i.e., intermediate viewpoint) is found for each display viewpoint. 
         [0114]    In step S 48 , the disparity image generating section  103  generates a recording source image (i.e., intermediate image data) whose viewpoint position is the position of the intersection point found for each display viewpoint. Then, in step S 49 , the disparity image generating section  103  generates a disparity image from the recording source image found in step S 48 , and the recording source image of the farther viewpoint from the display viewpoint. 
         [0115]    For example, as shown in  FIG. 11 , consider the case of generating the disparity images of a display viewpoint HS 21  and a display viewpoint HS 22  on the circle SC 11  with a radius d/2 centered at a reference position O 11 . 
         [0116]    It should be noted that in  FIG. 11 , portions corresponding to those in the case of  FIG. 10  are denoted by the same symbols, and description thereof is omitted as appropriate. Also, in the right side of  FIG. 11 , the portion in the vicinity of the display viewpoint HS 21  in the upper left in the drawing is shown in an enlarged fashion. 
         [0117]    First, when generating the disparity image of the display viewpoint HS 21 , in step S 46 , the disparity image generating section  103  selects the viewpoint VP 11  and the viewpoint VP 12  that are adjacent to the display viewpoint HS 21 . Then, the disparity image generating section  103  finds a straight line L 31  on the xy-plane which connects the viewpoint VP 12 , which is the nearer one to the display viewpoint HS 21  of these viewpoints VP 11  and VP 12 , and the reference position O 11 . It should be noted that the straight line L 31  can be also said to be a straight line connecting the viewpoint VP 6  located at a position opposed to the viewpoint VP 12  on the circle SC 11 , that is, the viewpoint VP 6  located on the side opposite to the viewpoint VP 12 , and the viewpoint VP 12 . 
         [0118]    Next, the disparity image generating section  103  finds a straight line L 32  that connects the viewpoint VP 11 , which is the farther one from the display viewpoint HS 21  of the viewpoint VP 11  and the viewpoint VP 12  adjacent to the display viewpoint HS 21 , and the display viewpoint HS 21 , and finds an intersection point IP 11  between this straight line L 32  and the straight line L 31 . This process of finding the intersection point IP 11  is the process in step S 47 . 
         [0119]    Further, the disparity image generating section  103  synthesizes the recording source image of the viewpoint VP 12 , which is located on the straight line L 31 , and the recording source image of the viewpoint VP 6  by a morphing method or the like, thereby generating a recording source image whose viewpoint position is the intersection point IP 11 . 
         [0120]    More specifically, the disparity image generating section  103  finds a distance RD from the viewpoint VP 12  to the intersection point IP 11 , and extrapolates the recording source image of the viewpoint VP 12  and the recording source image of the viewpoint VP 6  at a ratio between the distance RD and the distance d from the viewpoint VP 12  to the viewpoint VP 6 , that is, at a ratio of d:RD. This process of generating the recording source image whose viewpoint position is the intersection point IP 11  is the process in step S 48 . 
         [0121]    Further, the disparity image generating section  103  synthesizes the recording source image of the viewpoint VP 11 , and the recording source image whose viewpoint position is the intersection point IP 11  by a morphing method or the like, thereby generating the disparity image of the display viewpoint HS 21 . 
         [0122]    More specifically, the disparity image generating section  103  finds a distance DX 11  from the viewpoint VP 11  to the display viewpoint HS 21 , and a distance DX 12  from the display viewpoint HS 21  to the intersection point IP 11 . Then, the disparity image generating section  103  interpolates the recording source image of the intersection point IP 11 , and the recording source image of the viewpoint VP 11  at a ratio of DX 11 :DX 12 , thereby generating the disparity image of the display viewpoint HS 21 . This process of generating the disparity image is the process in step S 49 . 
         [0123]    It should be noted that the disparity image of the display viewpoint HS 22  is also generated in the same manner as the disparity image of the display viewpoint HS 21 . When disparity images are generated from recording source images whose viewpoints are located in the vicinity of their display viewpoints in this way, it is possible to easily obtain disparity images that are closer to the actual image and whose viewpoints are the found display viewpoints. 
         [0124]    Also, in the example of  FIG. 11 , the description is directed to the case of generating a disparity image by synthesizing the recording source image of the viewpoint VP 11 , which is the farther from the display viewpoint HS 21 , and the recording source image of the intersection point IP 11  in the vicinity of the nearer viewpoint VP 12 . However, this should not be construed restrictively. The disparity image of the display viewpoint HS 21  may be generated by synthesizing the recording source image of the nearer viewpoint VP 12 , and the recording source image of a point corresponding to the intersection point IP 11  in the vicinity of the farther viewpoint VP 11 . 
         [0125]    It should be noted, however, that in this case, the position of the point corresponding to the intersection point IP 11  in the vicinity of the viewpoint VP 11  is located farther away from the circle SC 11  than the intersection point IP 11 . Thus, degradation of image quality, which occurs due to extrapolation at the time of generating a recording source image, becomes greater than that at the time of generating the recording source image of the intersection point IP 11 . For this reason, it is desirable to generate the disparity image of the display viewpoint HS 21  by using the viewpoint VP 11  that is the farther from the display viewpoint HS 21 , and the recording source image of the intersection point IP 11 . 
         [0126]    Returning to the description of the flowchart in  FIG. 9 , once disparity images are generated, thereafter, the processes in step S 50  to step S 52  are performed and the stereoscopic display process ends. Since these processes are the same as the processes in step S 15  to step S 17  in  FIG. 15 , description thereof is omitted. 
         [0127]    In this way, in the image processing apparatus  71 , recording source images are generated from acquired source images, and only the recording source images are recorded in advance. Thus, the recording capacity for images required for obtaining disparity images can be reduced. Also, in the image processing apparatus  71 , display viewpoints are found from the user&#39;s sight line direction, and the disparity images of the display viewpoints are generated by using the recording source images that have been recorded. Thus, it is possible to easily obtain disparity images with which a subject is observed in a stereoscopic manner by the user, irrespective of the viewpoint position of the user. 
         [0128]    It should be noted that in the case where recording source images that have been already recorded in the recording section  82  are used to generate disparity images, there is no need to generate recording source images. Thus, the processes in step S 41  to step S 43  in  FIG. 9  are not performed. 
         [0129]    Also, the foregoing description is directed to the case of performing compression of information by generating recording source images whose viewpoints are located on the circumference of the circle SC 11 , assuming that the disparity d as the distance of separation of user&#39;s eyes has a fixed value. However, the disparity d may be made variable. 
         [0130]    In such a case, for example, the disparity image generating section  103  can generate recording source images whose viewpoints are located outside the circle SC 11 , by extrapolation using the recording source images of the viewpoint VP 12  and the viewpoint VP 6  in  FIG. 11 . Thus, for example, recording source images of a viewpoint VP 12 ′ and a viewpoint VP 6 ′ that are located outside the circle SC 11  are obtained. By using such recording source images, it is possible to obtain a pair of disparity images of display viewpoints located outside the circle SC 11 , with the disparity enhanced. 
         [0131]    That is, in this case, the viewpoint position determining section  102  draws a circle SC 11 ′ whose diameter equals a disparity d′ that has been changed, and whose center position (reference position O 11 ) is the same as the center position of the circle SC 11  before the change of the disparity d, and determines each new display viewpoint located on the circle SC 11 ′. By using recording source images recorded in the recording section  82 , the disparity image generating section  103  generates recording source images of the viewpoint VP 12 ′ and so on whose viewpoints are located on the circle SC 11 ′, and generates each disparity image after the change of the disparity, by using, among the obtained recording source images, recording source images whose viewpoints are located in the vicinity of the new display viewpoint. 
         [0132]    Likewise, the disparity image generating section  103  can also generate recording source images whose viewpoints are located inside the circumference of the circle SC 11 , by interpolation using the recording source images of the respective viewpoints VP 1  to VP 12  in  FIG. 11 . When recording source images obtained by interpolation are used, it is possible to obtain a pair of disparity images of display viewpoints located inside the circle SC 11 , with the disparity compressed. 
         [0133]    In this way, since the recording source images of arbitrary viewpoints can be obtained by interpolation and extrapolation, the disparity image generating section  103  can generate a disparity image pair with a specified, arbitrary disparity d, with respect to an arbitrary sight line direction. It should be noted that since interpolation generally causes less image degradation than extrapolation, when recording source images are generated while supposing the maximum circumference that can be obtained from source images whose viewpoints are arranged in an array, disparity images with better image quality can be obtained with respect to an arbitrary disparity d. 
       Third Embodiment 
       [0134]    &lt;Description of Stereoscopic Display Process&gt; 
         [0135]    Further, in the example of  FIG. 11 , the description is directed to the case of generating a recording source image whose viewpoint position is the intersection point IP 11 , and generating a disparity image by using the recording source image, and the recording source image of the viewpoint VP 11 . In this case, image synthesis needs to be performed twice to generate a single disparity image. Consequently, the image quality of the finally obtained disparity image may sometimes deteriorate. 
         [0136]    Accordingly, a recording source image whose viewpoint is located in the vicinity of each display viewpoint may be used to approximately generate the disparity image of the display viewpoint. Hereinbelow, referring to the flowchart in  FIG. 12 , a description will be given of a stereoscopic display process, which is performed in the case when the image processing apparatus  71  in  FIG. 8  approximately generates the disparity image of each display viewpoint. 
         [0137]    It should be noted that since the processes in step S 81  to step S 86  are the same as the processes in step S 41  to step S 46  in  FIG. 9 , description thereof is omitted. 
         [0138]    In step S 87 , the disparity image generating section  103  generates disparity images by using recording source images read from the recording section  82 . 
         [0139]    For example, as shown in  FIG. 13 , consider the case of generating the disparity images of the display viewpoint HS 21  and the display viewpoint HS 22 . It should be noted that in  FIG. 13 , portions corresponding to those in the case of  FIG. 11  are denoted by the same symbols, and description thereof is omitted. Also, in the right side of  FIG. 13 , the portion in the vicinity of the display viewpoint HS 21  in the upper left in the drawing is shown in an enlarged fashion. 
         [0140]    For example, suppose that, to generate the disparity image of the display viewpoint HS 21 , the recording source images of the viewpoint VP 11  and the viewpoint VP 12  which are adjacent to and on both sides of the display viewpoint HS 21  on the circle SC 11  are selected. That is, suppose that the disparity image is generated by using the recording source images of the viewpoint VP 11  and the viewpoint VP 12 . 
         [0141]    In this case, the disparity image generating section  103  defines, as a display viewpoint HS 21 ′, the position of the intersection point between a straight line L 51  connecting these viewpoints VP 11  and VP 12 , and a straight line L 52  that passes through the reference position O 11 . 
         [0142]    Then, the disparity image generating section  103  approximately regards the display viewpoint HS 21 ′ as the display viewpoint HS 21 , generates a disparity image whose viewpoint position is the display viewpoint HS 21 ′ from the recording source images of the viewpoint VP 11  and the viewpoint VP 12 , and defines the obtained disparity image as the disparity image of the display viewpoint HS 21 . That is, the arc of the circle SC 11  from the viewpoint VP 11  to the viewpoint VP 12  is regarded as a straight line, and the actual display viewpoint HS 21  is substituted by the display viewpoint HS 21 ′. 
         [0143]    The disparity image of the display viewpoint HS 21 ′ is generated by synthesizing the recording source image of the VP 11 , and the recording source image of the viewpoint VP 12  by a morphing method or the like. 
         [0144]    More specifically, for example, the disparity image generating section  103  finds an angle phi that is formed by a straight line connecting the viewpoint VP 12  and the reference position O 11 , and a straight line connecting the viewpoint VP 11  and the reference position O 11 , and an angle theta that is formed by the straight line connecting the viewpoint VP 12  and the reference position O 11 , and the straight line L 52 . Then, the disparity image generating section  103  interpolates the recording source image of the viewpoint VP 12  and the recording source image of the viewpoint VP 11  at the ratio between an angle (phi-theta) and the angle theta, that is, at the ratio of (phi-theta):theta, thereby generating the disparity image of the display viewpoint HS 21 ′. 
         [0145]    It should be noted that the disparity image of the display viewpoint HS 22  is also generated in the same manner as the disparity image of the display viewpoint HS 21 . When disparity images are generated by synthesizing recording source images whose viewpoints are located in the vicinity of their display viewpoints in this way, it is possible to easily obtain disparity images that are closer to the actual image and whose viewpoints are the found display viewpoints. 
         [0146]    Returning to the description of the flowchart in  FIG. 12 , once disparity images are generated, thereafter, the processes in step S 88  to step S 90  are performed and the stereoscopic display process ends. Since these processes are the same as the processes in step S 50  to step S 52  in  FIG. 9 , description thereof is omitted. 
         [0147]    In this way, the image processing apparatus  71  finds display viewpoints from the user&#39;s sight line direction, and synthesizes recording source images whose viewpoints are located in the vicinity of the display viewpoints, thereby approximately generating disparity images whose viewpoints are the display viewpoints. Thus, it is possible to easily and quickly obtain disparity images with which a subject is observed in a stereoscopic manner by the user, irrespective of the viewpoint position of the user. 
       Fourth Embodiment 
       [0148]    &lt;Configuration of Image Processing Apparatus&gt; 
         [0149]    It should be noted that in the example described above, a disparity image pair is displayed stereoscopically by the liquid crystal shutter scheme. Thus, if the sight line direction is substantially parallel to the xy-plane, the user can observe a subject in a stereoscopic manner no matter from which direction the user sees the display section  41 . 
         [0150]    However, in the case where a stereoscopic image is displayed by the parallax barrier scheme, the lenticular scheme, the polarization scheme, or the like, for the subject to be seen in a stereoscopic manner at the time when observing the stereoscopic image, the user must see the stereoscopic image from a specific direction (hereinafter, referred to as viewing direction). Accordingly, in the case where the user&#39;s viewing direction is restricted, the user may be allowed to specify the sight line direction with respect to the subject in an arbitrary fashion, so that the user can observe the stereoscopic image of the subject seen from the specified sight line direction. 
         [0151]    In such a case, the image processing apparatus  71  is configured as shown in  FIG. 14 , for example. It should be noted that in  FIG. 14 , portions corresponding to those in the case of  FIG. 4  are denoted by the same symbols, and description thereof is omitted as appropriate. 
         [0152]    The image processing apparatus  71  in  FIG. 14  includes the acquisition section  81 , the recording section  82 , the image processing section  84 , the display control section  85 , an input section  171 , and a display section  172 . 
         [0153]    The input section  171  is formed by an input device such as a dial or a mouse, for example, and is operated in cases such as when inputting the sight line direction. That is, the input section  171  accepts an input of the sight line direction or the like. 
         [0154]    The image processing section  84  includes the viewpoint position determining section  102 , the disparity image generating section  103 , and a rotation processing section  181 . The viewpoint position determining section  102  determines display viewpoints of disparity images on the xy-plane on the basis of the sight line direction supplied from the input section  171 , and supplies the display viewpoints to the disparity image generating section  103 . 
         [0155]    The disparity image generating section  103  reads source images from the recording section  82 , and generates the disparity images of the display viewpoints supplied from the viewpoint position determining section  102 , by using the read source images. Also, the disparity image generating section  103  supplies the generated disparity images to the rotation processing section  181 . The rotation processing section  181  rotates the disparity images supplied from the disparity image generating section  103 , by an angle defined by the sight line direction supplied from the input section  171 , and supplies the disparity images to the display control section  85 . 
         [0156]    The display section  172  is installed in such a way that the display screen is in a direction perpendicular to the vertical direction, that is, in a direction parallel to the xy-plane. The display section  172  displays the disparity images supplied from the display control section  85  by a predetermined display scheme such as the parallax barrier scheme, the lenticular scheme, or the polarization scheme. 
         [0157]    Here, as in the case of the display section  41 , the positional relationship between the center of the display screen of the display section  172 , and the viewpoint of each source image on the xyz-space is supposed to be the same as the positional relationship between the subject of the source images and the viewpoint of each source image. 
         [0158]    It should be noted that the display scheme for disparity images in the display section  172  may be any display scheme as long as the display scheme allows a subject on a stereoscopic image to be seen in a stereoscopic manner when the user observes the stereoscopic image from a specific direction. Also, in the following, it is supposed that the display section  172  displays a stereoscopic image in such a way that allows the user to see a subject on the stereoscopic image in a stereoscopic manner when the user&#39;s viewing direction is substantially parallel to the y-direction. 
         [0159]    Further, while the following description is directed to the case in which the display screen of the display section  172  is in a direction parallel to the xy-plane, the installation direction of the display screen of the display section  172  may be any direction. 
         [0160]    &lt;Description of Stereoscopic Display Process&gt; 
         [0161]    Next, referring to the flowchart in  FIG. 15 , a stereoscopic display process performed by the image processing apparatus  71  in  FIG. 14  will be described. It should be noted that since the processes in step S 111  to step S 113  are the same as the processes in step S 12  to step S 14  in  FIG. 5 , description thereof is omitted. 
         [0162]    It should be noted that in step S 111 , the viewpoint position determining section  102  determines a pair of display viewpoints on the basis of the sight line direction supplied from the input section  171 . Also, generated disparity images are supplied from the disparity image generating section  103  to the rotation processing section  181 . 
         [0163]    In step S 114 , the rotation processing section  181  rotates the disparity images supplied from the disparity image generating section  103 , in accordance with the sight line direction supplied from the input section  171 , and supplies the disparity images to the display control section  85 . 
         [0164]    For example, suppose that in the example shown in  FIG. 6 , the sight line direction SD 11  is inputted through operation of the input section  171 , and the disparity images of the display viewpoint HS 11  and the display viewpoint HS 12  are generated. Also, suppose that the user&#39;s viewing direction is a direction from the lower side to the upper side in the drawing. 
         [0165]    In this example, since the user&#39;s viewing direction is parallel to the y-direction, in order for the user to be able to perceive the subject in a stereoscopic manner, the disparity image pair must have a disparity in the x-direction. On the other hand, since the angle formed by the sight line direction and the x-direction is theta, it follows that the actual direction of the disparity of the disparity image pair is inclined by an angle ((pi/2)−theta) with respect to the x-direction. 
         [0166]    Accordingly, the rotation processing section  181  rotates the disparity images clockwise by the angle ((pi/2)−theta) in  FIG. 6 , and supplies the rotated disparity image pair as the final disparity image pair to the display control section  85 . By rotating disparity images by the amount of inclination of the disparity direction in this way, the rotated disparity image pair has a disparity in the x-direction. Thus, it is possible to display the image of a subject seen from an arbitrary direction in a stereoscopic manner, even in the case where the user&#39;s viewing direction is restricted. 
         [0167]    Returning to the description of the flowchart in  FIG. 15 , once the rotated disparity images are supplied to the display control section  85 , the processing proceeds from step S 114  to step S 115 . 
         [0168]    In step S 115 , the display control section supplies the disparity image pair supplied from the rotation processing section  181  to the display section  172 , and causes the disparity image pair to be displayed stereoscopically by the lenticular scheme or the like. Thus, the subject on the disparity images is seen in a stereoscopic manner by the user who is seeing the display section  172  with naked eyes. That is, a stereoscopic image is displayed. 
         [0169]    In step S 116 , the viewpoint position determining section  102  judges whether or not the user&#39;s sight line direction has changed. For example, the sight line direction is judged to have changed when a new sight line direction is supplied from the input section  171 . 
         [0170]    If it is judged in step S 116  that the sight line direction has changed, the processing returns to step S 111 , and the above-described process is repeated. That is, on the basis of the newly inputted sight line direction, disparity images having a disparity in the x-direction are generated, and are displayed stereoscopically. 
         [0171]    On the other hand, if it is judged in step S 116  that the sight line direction has not changed, in step S 117 , the image processing apparatus  71  judges whether or not to end the process of displaying a stereoscopic image. If it is judged in step S 117  not to end the processing, the processing returns to step S 116 , and the above-described process is repeated. 
         [0172]    On the other hand, if it is judged in step S 117  to end the processing, the image processing apparatus  71  causes display of a stereoscopic image to end, and the stereoscopic display process ends. 
         [0173]    In the manner as described above, on the basis of the inputted sight line direction, the image processing apparatus  71  causes disparity images to be displayed while being rotated so that the disparity direction and the viewing direction are orthogonal to each other. By rotating disparity images in accordance with the inputted sight line direction in this way, even in the case where the viewing direction for a stereoscopic image is defined, the user can observe the subject in a stereoscopic manner from an arbitrary direction while freely changing the sight line direction. 
       Fifth Embodiment 
       [0174]    &lt;Configuration of Image Processing Apparatus&gt; 
         [0175]    Also, in the case where disparity images are displayed stereoscopically by the lenticular scheme or the like as well, as in the case of the image processing apparatus  71  shown in  FIG. 8 , the amount of data of images to be recorded in advance may be reduced by generating recording source images from source images. 
         [0176]    In such a case, the image processing apparatus  71  is configured as shown in  FIG. 16 , for example. It should be noted that in  FIG. 16 , portions corresponding to those in the case of  FIG. 8  or  FIG. 14  are denoted by the same symbols, and description thereof is omitted as appropriate. 
         [0177]    The image processing apparatus  71  in  FIG. 16  is configured such that the recording source image generating section  131  shown in  FIG. 8  is further provided to the image processing apparatus in  FIG. 14 . That is, in the recording source image generating section  131 , recording source images are generated from source images acquired by the acquisition section  81 , and the generated recording source images are recorded to the recording section  82 . 
         [0178]    &lt;Description of Stereoscopic Display Process&gt; 
         [0179]    Next, referring to the flowchart in  FIG. 17 , a stereoscopic display process performed by the image processing apparatus  71  in  FIG. 16  will be described. 
         [0180]    It should be noted that in step S 141  to step S 143 , the same processes as the processes in step S 41  to step S 43  in  FIG. 9  are performed. That is, points on the circumference of a circle centered at a predetermined reference position on the xy-plane are defined as viewpoint positions of recording source images, and from one or a plurality of source images whose viewpoints are located in the vicinity of each of the viewpoints, recording source images are generated, and are recorded to the recording section  82 . 
         [0181]    Then, thereafter, the processes in step S 144  to step S 148  are performed. It should be noted that since these processes are the same as the processes in step S 45  to step S 49  in  FIG. 9 , detailed description thereof is omitted. 
         [0182]    It should be noted, however, that in step S 144 , the viewpoint position determining section  102  determines display viewpoints of disparity images from the sight line direction supplied from the input section  171  and the disparity d that is specified. Also, the disparity images of the display viewpoints are generated through the process described with reference to  FIG. 11 . The disparity images generated in this way are supplied from the disparity image generating section  103  to the rotation processing section  181 . 
         [0183]    In step S 149 , the rotation processing section  181  rotates the disparity images supplied from the disparity image generating section  103 , on the basis of the sight line direction supplied from the input section  171 , and supplies the disparity images to the display control section  85 . That is, in step S 149 , the same process as that in step S 114  in  FIG. 15  is performed. 
         [0184]    Once the rotated disparity images are generated, thereafter, the processes in step S 150  to step S 152  are performed and the stereoscopic display process ends. Since these processes are the same as the processes in step S 115  to step S 117  in  FIG. 15 , description thereof is omitted. 
         [0185]    In this way, in the case where disparity images are displayed by the lenticular scheme or the like as well, when recording source images are generated from source images and recorded in advance, the recording capacity for images required for generating the disparity images can be reduced. 
       Sixth Embodiment 
       [0186]    &lt;Description of Stereoscopic Display Process&gt; 
         [0187]    Further, the image processing apparatus  71  in  FIG. 16  may generate disparity images by performing the same process as the process described with reference to  FIG. 13 . Hereinbelow, referring to the flowchart in  FIG. 18 , a description will be given of a stereoscopic display process which is performed in the case when the image processing apparatus  71  in  FIG. 16  approximately generates disparity images of display viewpoints. 
         [0188]    It should be noted that since the processes in step S 181  to step S 184  are the same as the processes in step S 141  to step S 144  in  FIG. 17 , description thereof is omitted. 
         [0189]    In step S 185 , for each of display viewpoints supplied from the viewpoint position determining section  102 , the disparity image generating section  103  selects recording source images whose viewpoints are located in the vicinity of the display viewpoint, and reads the selected recording source images from the recording section  82 . 
         [0190]    For example, as shown in  FIG. 13 , the recording source images of the viewpoint VP 11  and the viewpoint VP 12  which are adjacent to and on both sides of the display viewpoint HS 21 , and the recording source images of the viewpoint VP 5  and the viewpoint VP 6  which are adjacent to and on both sides of the display viewpoint HS 22  are selected, and read. 
         [0191]    In step S 186 , the disparity image generating section  103  generates disparity images by using the read recording source images, and supplies the disparity images to the rotation processing section  181 . For example, in the example in  FIG. 13 , the recording source images of the viewpoint VP 11  and the viewpoint VP 12  are synthesized by a morphing method or the like into a disparity image, and likewise, the recording source images of the viewpoint VP 5  and the viewpoint VP 6  are synthesized by a morphing method or the like into a disparity image. 
         [0192]    Then, thereafter, the processes in step S 187  to step S 190  are performed and the stereoscopic display process ends. Since these processes are the same as the processes in step S 149  to step S 152  in  FIG. 17 , description thereof is omitted. That is, the obtained disparity images are rotated in accordance with the sight line direction and displayed stereoscopically. 
         [0193]    In the manner as described above, by finding display viewpoints from the inputted sight line direction, and synthesizing recording source images whose viewpoints are located in the vicinity of the display viewpoints to thereby approximately generate disparity images whose viewpoints are the display viewpoints, it is possible to easily and quickly obtain disparity images with which a subject is observed in a stereoscopic manner by the user. 
         [0194]    The series of processes described above can be either executed by hardware or executed by software. If the series of processes is to be executed by software, a program constituting the software is installed into a computer embedded in dedicated hardware, or into, for example, a general-purpose personal computer or the like that can execute various kinds of function when installed with various kinds of program, from a program-recording medium (i.e., a non-transitory, computer-readable storage medium). 
         [0195]      FIG. 19  is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes. 
         [0196]    In the computer, a CPU (Central Processing Unit)  301 , a ROM (Read Only Memory)  302 , and a RAM (Random Access Memory)  303  are connected to each other via a bus  304 . 
         [0197]    The bus  304  is further connected with an input/output interface  305 . The input/output interface  305  is connected with an input section  306  formed by a keyboard, a mouse, a microphone, or the like, an output section  307  formed by a display, a speaker, or the like, a recording section  308  formed by a hard disk, a non-volatile memory, or the like, a communication section  309  formed by a network interface or the like, and a drive  310  for driving a removable medium  311  such as a magnetic disc, an optical disc, a magneto-optical disc, or a semiconductor memory. 
         [0198]    In the computer configured as described above, the above-mentioned series of processes is performed when the CPU  301  loads a program recorded in the recording section  308  into the RAM  303  via the input/output interface  305  and the bus  304 , and executes the program, for example. 
         [0199]    The program executed by the computer (CPU  301 ) is provided by being recorded on the removable medium  311  that is a packaged medium formed by, for example, a magnetic disc (including a flexible disc), an optical disc (such as a CD-ROM (Compact Disc-Read Only Memory) or a DVD (Digital Versatile Disc)), a magneto-optical disc, a semiconductor memory, or the like, or via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcast. 
         [0200]    Then, the program can be installed into the recording section  308  via the input/output interface  305 , by mounting the removable medium  311  in the drive  310 . Also, the program can be received by the communication section  309  via a wired or wireless transmission medium, and installed into the recording section  308 . Alternatively, the program can be pre-installed into the ROM  302  or the recording section  308 . 
         [0201]    It should be noted that the program executed by the computer may be a program in which processes are performed in a time-series fashion in the order as described in this specification, or may be a program in which processes are performed at necessary timing such as when invoked. 
         [0202]    It should be noted that embodiments of the present invention are not limited to the above-described embodiments, and various modifications are possible without departing from the scope of the present invention. 
       REFERENCE SIGNS LIST 
       [0000]    
       
           41  display section 
           71  image processing apparatus 
           83  sensor 
           84  image processing section 
           101  sight line direction determining section 
           102  viewpoint position determining section 
           103  disparity image generating section 
           131  recording source image generating section 
           181  rotation processing section
An apparatus for processing source images having known viewpoints is disclosed. The apparatus may include a viewpoint determining section for determining, relative to the known viewpoints, left-eye and right-eye viewpoints of a viewer. Additionally, the FIG. 6 apparatus may include a disparity image generating section. The disparity generating section may be for selecting data of at least one of the source images, based on the determined left-eye viewpoint, as raw left-eye data. In addition, the disparity generating section may be for selecting data of at least one of the source images, based on the determined right-eye viewpoint, as raw right-eye data. The disparity generating section may also be for outputting left-eye image data, based on the raw left-eye data. Additionally, the disparity generating section may be for outputting right-eye image data, based on the raw right-eye data.