Safely promoting master to slave
When you shifted the IP within your application, any DB Connections that were open at the moment were totally unaware of the move. A quick netstat | grep -i mysql would reveal that on Master1.

You would be better off doing the cutover of the IP as follows:


On Master1

FLUSH HOSTS;
service mysql stop

Change the IP in the App
On all web servers, service httpd restart


In order to safeguard the app from having to edit it for the sake of assigning a new IP, try using a DB VIP instead.

For example:


Master1 is 10.1.2.30
Master2 is 10.1.2.40
Use 10.1.2.70 as the DBVIP


Here is what you can setup


On Master1, run ip addr add 10.1.2.70/24 dev eth1
Use 10.1.2.70 in your app


Then, when it is time to cutover, do the following


On Master1

FLUSH HOSTS;
service mysql stop
ip addr del 10.1.2.70/24 dev eth1

On Master2

ip addr add 10.1.2.70/24 dev eth1
SHOW SLAVE STATUS\G and make sure all final SQL statements from Master1 executed

On all web servers, service httpd restart


That way, a cutover would not involve editing any part of the app. Please notice that I did not mention running STOP SLAVE anywhere because this would allow any final SQL statements to flow over from Master1 to Master2 once mysql is stopped on Master1.