What's the difference between setting negative label as 0 or -1 in binary classification?
This will depend on the algorithm/code that you are using.
You are right that this is often not an important difference but that doesn't mean it can't create errors if you are using code which expects one or the other without being aware of it.
You should equally keep track of data types such as integers vs floats which will similarly produce different outcomes in lots of data science tools such as in Python or R.
Since lots of classifiers implemented in packages like sklearn will specify whether they are for continuous/discrete outcome spaces (classification vs regression), you shouldn't have to worry about this.
To illustrate this for sklearn:

source: https://scikit-learn.org/stable/_static/ml_map.png