1. An image processing apparatus for outputting an image to be displayed by a display unit, the apparatus comprising: a holding unit configured to hold a refocus image generated by performing image processing on multi-view image data indicating a plurality of images obtained by performing image-capturing from a plurality of different viewpoints, wherein the refocus image is an image with a focus adjusted from a focus of an image indicated by the multi-view image data after the image-capturing; a setting unit configured to set a refocus mode indicating a kind of target to be focused in a refocus image to be displayed by the display unit; and an outputting unit configured to read from the holding unit and output to the display unit, a refocus image which is focused on a target indicated by the refocus mode set by the setting unit, wherein the outputting unit, in a case where a user instruction to switch an image displayed by the display unit from a first refocus image corresponding to a first imaging scene to a second refocus image corresponding to a second imaging scene different from the first imaging scene is input and a user instruction to change the refocus mode set by the setting unit is not input, outputs, as the second refocus image, a refocus image focused on a target indicated by the same refocus mode as the first refocus image to the display unit.