1. A defect image classification apparatus comprising: an image acquisition unit which acquires images corresponding to a plurality of detectors, on the basis of signals obtained by detecting secondary particles obtained by emitting charged particle beams to a sample by the plurality of detectors; an automatic defect classification processing unit which automatically classifies kinds of defects included in the images, on the basis of the images; a display unit which displays the automatically classified kinds of the defects and images used for the automatic classification; a manual defect classification processing unit which classifies the kinds of the defects included in the images, according to an instruction input by a user, and modifies an automatic classification result; a control unit which selects images obtained from at least some detectors among the plurality of detectors, associated with the kinds of the defects to be the classification result of the automatic defect classification processing unit, as images displayed initially on the display unit when the instruction is input by the user; and an image switching instruction input unit which receives an input of the user showing switching the images displayed on the display unit into images obtained by other detectors at the same time as acquisition of the images, wherein the control unit calculates an appearance frequency of image kinds at a time of determining a defect kind, on the basis of a switching operation log of the user for the image switching instruction input unit when images of defects determined previously as the same kinds as the kinds of the defects determined by the automatic defect classification processing unit are classified in accordance with an instruction from the user; and the control unit associates the kinds of the defects and the images displayed initially on the display unit, based on the appearance frequency.