Can a single bad letter of recommendation ruin my chances of getting admitted to grad school?
Pick someone else if it isn't already too late.

The actual answer to your question is this: It is far better to have a good letter from a faculty member who is relatively unknown than a bad letter from someone they recognize and respect.  In fact, the latter is probably the worst thing that could happen.

If it is already too late, i.e. you have asked Professor X, he's agreed, and you've sent him the material (which is probably an online link these days) then work hard to make writing the letter easy.  Put together the following information and send it fast:


Include your student number.
Remind him which of his classes you have taken, and when.
How did you distinguish yourself in those classes?
How would you describe yourself? What are your strengths? What are your weaknesses? He is going to have to answer those questions when he writes your reference, so the more details the better; but these must be things your referee knows himself.
What are some of your academic and nonacademic accomplishments that he may not remember?
What makes him particularly qualified to write a letter for you? That is, why should the recipient of the letter value it over a letter from someone else?


This goes in an email that says, "Thank you very much for agreeing to write a reference for me on very short notice.  I hope the following will be of use to you when you write it."