Early career advice for Jr. Researcher on selectivity of authorship credit for collaboration in areas outside one's specialization
I don't think that you're too picky or that you should relax your standards. 

If you are responsible for creating a statistical model to test the data, then you should know everything there is to know about the process that generated the data that you are modeling. If you do not know everything and if you don't have access to all the data and any material relevant to how the data was generated (e.g. protocols), then you cannot in good conscience model the data. The reason for this is that you don't know where to look for problems that might invalidate your model. You might, therefore, overlook something. And if the wrong statistical model is used, then the sampling distribution of the parameters and your test statistics will be wrong. They will be wrong because the premises of your argument will be false. 

Ideally, as the person conducting the statistical analysis, you will be involved in collecting the data. This means that you should be involved in designing the experiment. Fisher once said the following about involving a statistician after the fact: 


  "To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of."


In the case of an observational study this might not be as bad as you could collect other data after you're involved. But it is not ideal. 

What I would recommend you to do is to use the argument that you cannot possibly model the data if you don't have access to all the materials and know how the data were generated. You can use this argument to justify yourself to your co-workers. And you can use this argument to justify yourself towards yourself for not working on certain projects. You're the expert here and you need to tell yourself and others what you need to be able to do your work. And if what you need isn't there, then you can't do your work. Period. This way you avoid re-thinking, worrying, and arguing about what you should work on and what not.

In addition, I would recommend you to document everything for each paper to make it clear what your responsibilities were and what the responsibilities of other people were. In particular, in those cases where you did not have access to the raw data. 

As for the consequences of being part of a poorly conducted research. It will tarnish your reputation. By how much depends on what the norm is in your field and in your peer group. Your reputation is how you are judged by the standards of other people. As I am not in your field or know you (or what research you were involved in or would be), I cannot say how much of an impact this will have. But it will definitely be bad for your reputation. It might be worth it though. Although this depends on what your goals in life are.