This paper considers robust low-rank matrix completion in the presence of outliers. The objective is to recover a low-rank data matrix from a small number of noisy observations. We exploit the bilinear factorization formulation and develop a novel algorithm fully utilizing parallel computing resources. Our main contributions are i) providing two smooth loss functions that promote robustness against two types of outliers, namely, dense outliers drawn from some elliptical distribution and sparse spike-like outliers with small additive Gaussian noise; and ii) an efficient algorithm with provable convergence to a stationary solution based on a parallel update scheme. Numerical results show that the proposed algorithm obtains a better solution with faster convergence speed than the benchmark algorithms in both synthetic and real data scenarios.