What should I do if the first author wants me to write the whole paper?
First of all: Good call to talk about authorship issues at the beginning, rather than only once troubles started! Now authorship issues do occasionally have to be re-negotiated depending on the course the project takes.

There do seem to be potentially two separate aspects in your question:

A. What to do if I have contributed as much as I am willing as a secondary author, yet the first author keeps asking for more?
- Point out that you have done X, Y and Z, and that you are only willing to do significantly more if you are the first author (fundamental reading + commenting remain unaffected of course).

B. I started a side-project, and I no longer want to work on this AT ALL.
- Make clear to your co-author that you have no intention to work on this project anymore, and that this pertains to the future, too. Either give them free rein to continue the project without you (either with you on the author list or not, depending on your preferences); OR make sure you are still available for basic reading/commenting, but simply nothing more extensive.

In both cases, if the coauthor shows a significantly stronger reaction than "medium annoyance", then maintaining a positive relationship to them is futile anyway.