How to measure the information of covariates in a ML task?
There exists a notion of entropy in information theory. It is in fact closely linked to covariance, and especially the covariance matrix determinant (I don't recall the actual mathematical link but I think this should be something that Google can find).

From then, you could use some metrics based on the dataset correlation matrix. You could use its determinant (low determinant means highly correlated data), but it's not always easy to analyse (2 perfectly correlated features will drop the determinant value to 0). The distribution of correlation matrix eigenvalues will surely give more detailed insight (multiple low eigenvalues will mean lots of correlated features).

Another powerful and less mathematical solution would be to compute PCA over the dataset. Give yourself a threshold of explained variance (for instance 95%), and count how many principal components you need to reach this explained variance. A low number of such would mean low "quality" data.