I tried to merge 2 vertices, but it keeps moving to the center
This could be related to the mirror modifier. If the body was made on the right side and the horns on the left, they look ok, but the real vertices are on the opposite sides. Deactivate your mirror's visiblity to check this.  

You should work with the "Adjust Edit Cage" of the mirror modifier disabled to avoid creating things hard to merge.