Refresh Calendar.app with correct Exchange-account data
For me the solution was: 


Open in Finder by running open ~/Library/Calendars in Terminal
Look for a folder that ends with .exchange. Open it.
Find the correct Exchange calendar by looking at the Info.plist files in each of the .calendar folders. Open the Events folder.
If you know that the problem is recent, sort all files by Date Modified. Set Cover Flow mode for showing the files in Finder
By looking at the items in Finder, Skip through the ics-files to search for the item that has the wrong date.
"Move to Trash" the incorrect item
Start the Calendar.app


I guess that it might be possible to "Move to trash" the .calendar folder for a full refresh of everything, but for me it was not necessary.