Will once turning down Master's admission impact a PhD application in the same department
I think it would be highly unlikely that they would disadvantage you, unless you are applying to a truly small and petty department. The small part is really key here: for most departments of any size, you would likely be one among potentially dozens to hundreds of applicants. That they would even remember your name on a later admissions from an MS application would be unlikely, unless they already felt you were someone special (which wouldn't hurt you).