Saving Calendar entries when deleting exchange account
I am basing this on something I have done in earlier versions of Calendar, like when it was called "iCal" so YMMV.

You should be able to select the specific calendar in calendar.app and then export it to an .ical file. Then delete your exchange account (did it sync Address Books as well?) and then if it just deleted the calendar you can re-import it.