Add Google calendar reminder numbers as one contact
I think Google Calendar SMS reminders are supposed to come from only one number (or only a few numbers), which might be different from user to user. For example, you would get reminders from numbers A, B, and C, while I would get reminders from numbers D, E, and F. So, if you add A, B, and C under one contact, that should solve your problem.

However, there seems to be a recent issue causing reminders to be sent from many random numbers. The issue has been recognized by a Google employee (KatherineG), so hopefully they are working on a fix. You might want to head over to that thread and add your country / carrier to the list affected by this problem.