A stylized letter F. Eliminating dairy from your diet can improve skin texture and tone and improve skin conditions like acne and rosacea. Skim milk causes the most skin issues because of the whey protein, hormones, and sugars in it.