1. An object monitoring apparatus, comprising: an image receiver configured to receive at least one frame of captured images; an edge image generator configured to generate an edge image by detecting edges of objects appearing in the frame; a reference image generator configured to generate a reference image by detecting a part corresponding to a background in the frame to thereby define the detected part as a background edge, the reference image generated by repeatedly detecting as many as or more than a predetermined number an edge commonly appearing over a plurality of the frames; a candidate object extractor configured to extract one or more candidate object pixels by comparing the edge image with the reference image, and to extract a candidate object by grouping the extracted candidate object pixels into the candidate object; and an object-of-interest determiner configured to determine whether the candidate object is an object-of-interest based on a size of the candidate object and a duration time of detection of the candidate object.