Endmember extraction is one of the most important issues in hyperspectral image analysis. Under the linear mixing model and pure pixel assumption, a number of convex-geometry-based methods have been developed in the past decades. However, these traditional methods generally produce unsatisfactory results since they require the hyperspectral image to have a convex structure and this is not exactly the case with the real image scene. The particle swarm optimization (PSO) algorithm has recently been employed to address the endmember extraction problem, but its performance is limited by the premature convergence and lower precision of the standard PSO, and much effort is required to enhance the optimization result. To address these problems, in this study, a novel quantum-behaved particle swarm optimization (QPSO) algorithm is proposed for hyperspectral endmember extraction. The notable advantages of the proposed method include: 1) a row-column coding approach for the particles is designed to accelerate the optimization process; 2) a cooperative approach is employed to update the particles' individual and global best positions, in order to help the particles' optimization behavior in the multidimensional search space; and 3) two kinds of objective functions, namely, maximizing the simplex volume formed by the endmember combination, and minimizing the root-mean-square error between the original image and its remixed image, are incorporated in a sequential optimization approach for the endmember extraction problem, which makes the algorithm robust to outliers at an acceptable time cost. The extensive experimental results prove that QPSO is able to find the optimal endmember combination.