People with a lactose intolerance or a milk allergy often replace milk by a substitute . Lactose intolerance or a milk allergy can prompt the use of milk substitutes ; they are also consumed by vegans .