Can I include in my CV the invitation to a conference I'm not sure to attend?
I would say no.

I often turn down invitations that could be considered a distinction, i.e. an indication that somebody thinks I am reasonably competent at this research thing:


Will you speak on this panel at this research open house?
Are you willing to review this paper for this journal?
Would you want to do a research internship with us this summer?
Do you want to submit an invited paper to this workshop?
Can you mentor this undergrad who is doing research in the lab?


All of these things would go on my CV if I actually do them, but not if I turn down the invitation.

An invitation to speak at a conference is no different, in my opinion. It's certainly a nice validation of your work that the conference organizers recognized your ability to contribute. But if you don't actually give the talk, it doesn't go on the CV.