Should I do the factory reset? If yes, what will I lose?
Doing a factory reset will not modify the SD card in any way. Only the data on the phone will be deleted. Contacts that are set up to sync with your Google account will restore, but everything else will be gone. 

If you have applications installed on the SD card, I would start there. Uninstall them, or move them back to the device storage. You say that it only crashes when it is in your phone, and not any other Android phone. This would make me think there is probably an app installed on the SD card for your phone. You also said that it doesn't reboot when a different card is in your phone, again leading me to believe that there could be an app that is installed to the SD card that is causing problems.

If you think there is something in your data that is causing the issue, then backing up the data with Titanium Backup will just restore any "issue" that there is. 

If you think the problem is with the data on the SD card, then you can format it under storage in settings.

I would also consider the SD card itself could have issues. The SD cards that come with the devices are very cheap cards and usually one of the slower class type of SD card. I have had a few of them last only a couple of months before. I suggest getting a better SD card, one that is at least a class 6. If you have a slower class card, this could also potentially cause issues with some applications.