MySQL create index in 1B rows table We're trying to add an index to one of our tables in production database.

CREATE TABLE `abcd_s3_metadata` (
  `rowId` bigint(20) NOT NULL AUTO_INCREMENT,
  `sId` varchar(50) NOT NULL,
  `ruid` varchar(200) NOT NULL,
  `s3Pointer` varchar(250) NOT NULL,
  `updateTime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `latest` tinyint(1) NOT NULL DEFAULT '1',
  `version` int(11) NOT NULL,
  PRIMARY KEY (`rowId`),
  UNIQUE KEY `ruid_UNIQUE` (`ruid`)
)

The table has ~ 1B rows.
We created a copy of the live table and I tried to add an index on updateTime field. 
With the following statement:


ALTER TABLE `innodb`.`ruids_s3_metadata` 
ADD INDEX `updateTime_idx` (`updateTime` ASC);


The ALTER statement ran for 3 hours resulted in extra read 1500 IOPS consumed all free InnoDB memory. And eventually caused the DB to choke and become unusable.

What can we do better? Are there other options?

Thanks

Alex
You should look at using a 3rd party schema change tool to avoid causing issues. You can start with pt-online-schema-change from the Percona Toolkit and if that doesn't meet your needs there's a schema change tool written by github named Ghost. 

https://www.percona.com/doc/percona-toolkit/LATEST/pt-online-schema-change.html

https://github.com/github/gh-ost