Renaming SQL Server Agent operator
In order to change the name (or any other property of an operator), you'd make a call to msdb.dbo.sp_update_operator.  For instance:

exec msdb.dbo.sp_update_operator 
    @name = 'OldOperatorName', 
    @new_name = 'NewOperatorName';
go


As for SSMS:  most things you do in SSMS you have to option to script out the changes you are making in the GUI.  So, without hitting Ok, you can select the Script button at the top to see what T-SQL SSMS will be executing.  It seems as though even with a changed name (i.e. altered text in the Name text box), the generated T-SQL doesn't contain a specified @new_name parameter for the stored procedure call.

Moral of the story?  Don't rely on a UI.  If you want to have full control over what's happening then you should be using T-SQL.  There will be no question in your mind if you're writing out the code to do what you want.  No assumptions, nothing.

Regarding changing the name, SQL Server stores the ID of the operator to create the correlation between notification -> operator, and not the operator name.  So no, you are not bound to the operator name and I don't believe changing the name of the operator will break anything.