MySQL- Performance implications of changing a column collation from latin1_swedish_ci to latin1_bin
Not significant performance difference.

It will require rebuilding any indexes involving that column.  That will take time, but it is a onetime issue.

Any *_ci collation is slightly more complex than a *_bin collation, at least when doing comparisons or sorting.

If you choose to undo the change, be explicit about the collation.  Your proposed ALTER hopes that the 'default' will be used.

An ALTER .. COLUMN will rebuild the necessary indexes.