How can I remove the OSX server?
As an admin, open the /Applications folder, and drag the Server.app icon to the trash. An alert should pop up, saying the the server will be shut down, and a bunch of uninstall scripts will run. Your data will still be in /Library/Server.

You can empty the trash or pull it back out, if you want to keep it.

Then you should be able to upgrade the OS in peace.