What are the physical properties affected in ZnO when there is a sulfur impurity?
Of course, if $\ce{ZnO}$ has a very small $\ce{S}$ impurity level, this could be the reason of the yellow colour of products. 
As for the effects of very little sulfur on tons of $\ce{ZnO}$ dust, it all depends on the end use of $\ce{ZnO}$:The applications of zinc oxide powder are numerous. Most applications exploit the reactivity of the oxide as a precursor to other zinc compounds. For material science applications, zinc oxide has high refractive index, high thermal conductivity, binding, antibacterial and UV-protection properties. Consequently, it is added into materials and products including plastics, ceramics, glass, cement, rubber, lubricants, paints, ointments, adhesive, sealants, concrete manufacturing, pigments, foods, batteries, ferrites, fire retardants, etc...

Clearly, if $\ce{ZnO}$ is used in rubber industry, sulfur impurities have no harm effect.
If it is used in electrical applications (as a semiconductor), sulfur will affect the electrical conductivity. 
If it is used in coating and paints applications, sulfur will affect the the color ....