1. A moving object detection apparatus, comprising: motion vector calculation means for calculating motion vectors from an input image; motion vector removal means for removing a motion vector having high randomness from said calculated motion vectors; motion vector accumulation means for temporally accumulating each said motion vector not removed by said motion vector removal means, and for calculating an accumulated number of occurrences and an accumulated value of each motion vector; and moving object detection means for determining, based on said calculated accumulated value and calculated accumulated number of occurrences of each motion vector, whether each said motion vector corresponds to a moving object, wherein said motion vector accumulation means includes: accumulated value storage means for storing said accumulated values and numbers of said previous motion vectors such that said accumulated values and numbers are to be associated with the preceding positions of said motion vectors, accumulated value readout means for reading out said accumulated values and numbers of said previous motion vectors from said accumulated value storage means, and accumulated value update means for calculating new said accumulated values and numbers of said motion vectors by performing an update based on said accumulated values and numbers of said previous motion vectors and a magnitude and position of said current motion vector, said accumulated values and numbers of said previous motion vectors being stored in said accumulated value storage means, said accumulated value readout means selects, from among said accumulated values and numbers of said previous motion vectors that are associated a position which is spatially a distance equivalent to a magnitude of said current motion vector away from a location of said current motion vector in a direction opposite to said current motion vector or that are associated with vicinity of said position, said accumulated value and number of said motion vector.