HDFS rolling upgrade allows upgrading individual HDFS daemons. For examples, the datanodes can be upgraded independent of the namenodes. A namenode can be upgraded independent of the other namenodes. The namenodes can be upgraded independent of datanodes and journal nodes.
In Hadoop v2, HDFS supports highly-available (HA) namenode services and wire compatibility. These two capabilities make it feasible to upgrade HDFS without incurring HDFS downtime. In order to upgrade a HDFS cluster without downtime, the cluster must be setup with HA.
If there is any new feature which is enabled in new software release, may not work with old software release after upgrade. In such cases upgrade should be done by following steps.
Note that rolling upgrade is supported only from Hadoop-2.4.0 onwards.
In a HA cluster, there are two or more NameNodes (NNs), many DataNodes (DNs), a few JournalNodes (JNs) and a few ZooKeeperNodes (ZKNs). JNs is relatively stable and does not require upgrade when upgrading HDFS in most of the cases. In the rolling upgrade procedure described here, only NNs and DNs are considered but JNs and ZKNs are not. Upgrading JNs and ZKNs may incur cluster downtime.
Run “hdfs dfsadmin -rollingUpgrade prepare” to create a fsimage for rollback.
Run “hdfs dfsadmin -rollingUpgrade query” to check the status of the rollback image. Wait and re-run the command until the “Proceed with rolling upgrade” message is shown.
Start NN2 as standby with the “-rollingUpgrade started” option.
Failover from NN1 to NN2 so that NN2 becomes active and NN1 becomes standby.
Start NN1 as standby with the “-rollingUpgrade started” option.
Choose a small subset of datanodes (e.g. all datanodes under a particular rack).
Run “hdfs dfsadmin -shutdownDatanode <DATANODE_HOST:IPC_PORT> upgrade” to shutdown one of the chosen datanodes.
Run “hdfs dfsadmin -getDatanodeInfo <DATANODE_HOST:IPC_PORT>” to check and wait for the datanode to shutdown.
Upgrade and restart the datanode.
Perform the above steps for all the chosen datanodes in the subset in parallel.
Repeat the above steps until all datanodes in the cluster are upgraded.
Run “hdfs dfsadmin -rollingUpgrade finalize” to finalize the rolling upgrade.
In a federated cluster, there are multiple namespaces and a pair of active and standby NNs for each namespace. The procedure for upgrading a federated cluster is similar to upgrading a non-federated cluster except that Step 1 and Step 4 are performed on each namespace and Step 2 is performed on each pair of active and standby NNs, i.e.
For non-HA clusters, it is impossible to upgrade HDFS without downtime since it requires restarting the namenodes. However, datanodes can still be upgraded in a rolling manner.
Start NN with the “-rollingUpgrade started” option.
When the upgraded release is undesirable or, in some unlikely case, the upgrade fails (due to bugs in the newer release), administrators may choose to downgrade HDFS back to the pre-upgrade release, or rollback HDFS to the pre-upgrade release and the pre-upgrade state.
Note that downgrade can be done in a rolling fashion but rollback cannot. Rollback requires cluster downtime.
Note also that downgrade and rollback are possible only after a rolling upgrade is started and before the upgrade is terminated. An upgrade can be terminated by either finalize, downgrade or rollback. Therefore, it may not be possible to perform rollback after finalize or downgrade, or to perform downgrade after finalize.
Downgrade restores the software back to the pre-upgrade release and preserves the user data. Suppose time T is the rolling upgrade start time and the upgrade is terminated by downgrade. Then, the files created before or after T remain available in HDFS. The files deleted before or after T remain deleted in HDFS.
A newer release is downgradable to the pre-upgrade release only if both the namenode layout version and the datanode layout version are not changed between these two releases.
Downgrade and restart the datanode.
Repeat the above steps until all upgraded datanodes in the cluster are downgraded.
Run “hdfs dfsadmin -rollingUpgrade finalize” to finalize the rolling downgrade.
Note that the datanodes must be downgraded before downgrading the namenodes since protocols may be changed in a backward compatible manner but not forward compatible, i.e. old datanodes can talk to the new namenodes but not vice versa.
Shutdown all NNs and DNs.
Restore the pre-upgrade release in all machines.
Start NNs with the “-rollingUpgrade downgrade” option.
Rollback restores the software back to the pre-upgrade release but also reverts the user data back to the pre-upgrade state. Suppose time T is the rolling upgrade start time and the upgrade is terminated by rollback. The files created before T remain available in HDFS but the files created after T become unavailable. The files deleted before T remain deleted in HDFS but the files deleted after T are restored.
Start NN1 as Active with the “-rollingUpgrade rollback” option.
Run `-bootstrapStandby’ on NN2 and start it normally as standby.
Start DNs with the “-rollback” option.
Execute a rolling upgrade action.
query Query the current rolling upgrade status.
prepare Prepare a new rolling upgrade.
finalize Finalize the current rolling upgrade.
Get the information about the given datanode. This command can be used for checking if a datanode is alive like the Unix ping command.
Submit a shutdown request for the given datanode. If the optional upgrade argument is specified, clients accessing the datanode will be advised to wait for it to restart and the fast start-up mode will be enabled. When the restart does not happen in time, clients will timeout and ignore the datanode. In such case, the fast start-up mode will also be disabled.
Note that the command does not wait for the datanode shutdown to complete. The “dfsadmin -getDatanodeInfo” command can be used for checking if the datanode shutdown is completed.
When a rolling upgrade is in progress, the -rollingUpgrade namenode startup option is used to specify various rolling upgrade options.
downgrade Restores the namenode back to the pre-upgrade release and preserves the user data.
rollback Restores the namenode back to the pre-upgrade release but also reverts the user data back to the pre-upgrade state.
started Specifies a rolling upgrade already started so that the namenode should allow image directories with different layout versions during startup.