Going from co-advisors to just one advisor
I think the optimal answer depends mostly on your situation and university. However, I try to give you some general advice. From what I have seen in your question, you might already be one step to far into the decision. I would rather start an open discussion with your first advisor and express your concern. All further steps, such as switching advisors, can be the outcome of your discussion, but there might be different and better solutions.

It is quite normal that research interests shift after some time into your PhD. Since you are making progress, you see which ideas are working out and which don't. You will get to know new topics and new questions will arise. This will have an influence on your research, of course.

Your advisor seems to see that, too:


  But he recently mentioned he's no longer sure how to advise me anymore, so I think he might be thinking the same thing.


By expressing his concern, he already shows some awareness of this issue. You should try to talk to him and find a solution together. I would not approach him like "I want to replace you with advisor Y". This can come across as very rude since this is on a personal level. Start a discussion that focuses on your research. Talk about your concerns that your interest have drifted apart and try to come up with possible solutions. When you are further into discussion you could also include the co-advisor. As already mentioned, switching advisors might be one solution but it might not be the best one.

The more you integrate your advisors into the discussion, the less conflicts I would expect. Do not push your decisions onto them as this could really affect your current position. Talk to them and find a solution together.