Is a correlation matrix meaningful for a binary classification task?
Well correlation, namely Pearson coefficient, is built for continuous data. Thus when applied to binary/categorical data, you will obtain measure of a relationship which does not have to be correct and/or precise.

There are quite a few answers on stats exchange covering this topic - this or this for example.