For the same reason as unilateral disarmament is foolish between nations, unilateral colorblindness is foolish between races.

If you don't BOTH get rid of your arms the one still armed is going to win.  That basic fact is what we've seen.