How to determine whether a bad performance is caused by data quality?
First, it sounds like your choice of model selection is a problem here.  Your outputs are binary-valued, not continuous. Specifically you may have a classification problem on your hands rather than a traditional regression problem.  My first recommendation would be to try a simple classification approach such as logistic regression or linear discriminant analysis.

Regarding your suspicions of bad data, what would bad data look like in this situation? Do you have reason to suspect that your $X$ values are noisy or that your $y$ values are mislabeled? It is also possible that there is not a strong relationship between any of your features and your targets.  Since your targets are binary, you should look at histograms of each of your features to get a rough sense of the class conditional distributions, i.e. $p(X_1|y=1)$ vs $p(X_1|y=0)$. In general though, you will need to be more specific about what "bad data" means to you.