Do variant callers miss rare variants in reference?
If the reference sequence is associated with a disease or increased susceptibility to some phenotype, the the population you're using to compare against (with a variant) will show protection to it. So either way you're going to find an association, this will just change the sign of the coefficient.