Diffusion process in p-n junction
This is pulled from the Wiki : Depletion Region in a PN Junction

diffused electrons come into contact with holes on the P-side and are eliminated by recombination. Likewise for the diffused holes on the N-side. The net result is the diffused electrons and holes are gone, leaving behind the charged ions adjacent to the interface in a region with no mobile carriers (called the depletion region). The uncompensated ions are positive on the N side and negative on the P side. This creates an electric field that provides a force opposing the continued exchange of charge carriers. When the electric field is sufficient to arrest further transfer of holes and electrons, the depletion region has reached its equilibrium dimensions.

Both carriers take part in the diffusion process. The diffusion is halted by the electric potential formed at the junction as a result of the process.