Should I use recommendation letter from young professor I worked with or well-known professor I met in class?
Your REU evidence that X wrote you a weak letter is not very strong.  I run an REU, and we accept less than 3% of our applicants.  Our target participants are either legendary at small colleges (best in a decade), or excellent at top universities (top 25% this year at Harvard).  Plenty of excellent people get turned away for essentially random reasons.  Maybe one of your letters was submitted late.  Maybe you forgot to list a course on your application that was considered essential that year.  Maybe I had a bad burrito for lunch and hated your essay when I read it.  Maybe there were just lots of really good applicants that year.  And yes, maybe professor X (or professor Z, the other letter-writer) said or implied something negative about you.

In addition, Professor Y offering to write on your behalf after a one month course says more about her than it does about you.  This does not guarantee that she will write a strong letter, and if she does she may write strong letters for a lot of people, which might be known by the people reading her letters.

Here is my advice.  Go to Professor Y and tell her that you did research with Professor X and published a paper together.  Then, ask her whether you should ask for X's recommendation or hers.  If she really feels that you are spectacular, she will insist on writing the letter herself.