Is it really the responsibility of a conference organizer to provide the necessary letters to the participants for their visa processing?
Is it their responsibility?

Yes. I've run into this issue before, but I have never had a conference organizer demur on writing the letters. Not only is it the responsibility of the conference organizer, strictly speaking they are the only person who can provide you with the appropriate letters, since the letter is the justification for your attendance. Someone else cannot write you a letter saying "Tod is attending because of some other conference."

What should you do?

Don't go. On the one hand this is a moot point, because if they don't write the letter, you won't have the visa. On the other hand, the unwillingness to write a letter leads me to believe that this is not a genuine conference. There is an industry of essentially fraudulent journals and conferences. Many of the conferences are held in places like China, I suppose so that people have an opportunity to travel there and because it is relatively cheap, which require letters to travel. Not getting a letter is a huge warning sign.

It is undoubtedly disappointing, but if the conference is not going to do the absolute minimum required, and it sounds like you are paying, you are almost assuredly better off not attending.