title: Hortonworks HA Namenodes gives an error "Operation category READ is not supported in state standby"
body: <p>My hadoop cluster HA active namenode (host1) suddenly switch to standby namenode(host2). I could not found any error in hadoop logs (in any server) to identify the root cause.</p>
  
  <p>After switching the Namenodes following error appeared in hdfs logs frequently and non of the application could read the HDFS files.</p>
  
  <blockquote>
    <p>2014-07-17 01:58:53,381 WARN  namenode.FSNamesystem
    (FSNamesystem.java:getCorruptFiles(6769)) - Get corrupt file blocks
    returned error: Operation category READ is not supported in state
    standby</p>
  </blockquote>
  
  <p>Once I restart the new active node(host2), namenode is switching back to new standby node(host1). Then cluster is working as normal, users also can can retrieve the HDFS files.</p>
  
  <p>I'm using Hortonworks 2.1.2.0 and HDFS version 2.4.0.2.1 </p>
  
  <p><strong>Edit:21st Jult 2014</strong> 
  Following logs were found in active namenode logs when active-standby namenode switch happen</p>
  
  <blockquote>
    <p>NT_SETTINGS-1675610.csv      dst=null        perm=null 2014-07-20
    09:06:44,746 INFO  FSNamesystem.audit
    (FSNamesystem.java:logAuditMessage(7755)) - allowed=true<br>
    ugi=storm (auth:SIMPLE) ip=/10.0.1.50   cmd=getfileinfo
    src=/user/tungsten/staging/LEAPSET/PRODUCTS/PRODUCTS-138018
    6.csv        dst=null        perm=null 2014-07-20 09:06:44,747 INFO  FSNamesystem.audit (FSNamesystem.java:logAuditMessage(7755)) -
    allowed=true       ugi=storm (auth:SIMPLE) ip=/10.0.1.50<br>
    cmd=getfileinfo
    src=/user/tungsten/staging/LEAPSET/MERCHANT_SETTINGS/MERCHA
    NT_SETTINGS-1695794.csv      dst=null        perm=null 2014-07-20
    09:06:44,747 INFO  FSNamesystem.audit
    (FSNamesystem.java:logAuditMessage(7755)) - allowed=true<br>
    ugi=storm (auth:SIMPLE) ip=/10.0.1.50   cmd=getfileinfo
    src=/user/tungsten/staging/LEAPSET/PRODUCTS/PRODUCTS-139954
    1.csv        dst=null        perm=null 2014-07-20 09:06:44,748 INFO  namenode.FSNamesystem (FSNamesystem.java:stopActiveServices(1095)) -
    Stopping services started for active state 2014-07-20 09:06:44,750
    INFO  namenode.FSEditLog (FSEditLog.java:endCurrentLogSegment(1153)) -
    Ending log segment 842249 2014-07-20 09:06:44,752 INFO 
    namenode.FSEditLog (FSEditLog.java:printStatistics(673)) - Number of
    transactions: 2 Total time for transactions(ms): 0 Number of
    transactions batched in Syncs: 0 Number of syncs: 1 SyncTimes(ms):  4
    35  2014-07-20 09:06:44,774 INFO  namenode.FSEditLog
    (FSEditLog.java:printStatistics(673)) - Number of transactions: 2
    Total time for transactions(ms): 0 Number of transactions batched in
    Syncs: 0 Number of syncs: 2 SyncTimes(ms):  24 37  2014-07-20
    09:06:44,805 INFO  namenode.FSNamesystem (FSNamesystem.java:run(4362))
    - NameNodeEditLogRoller was interrupted, exiting 2014-07-20 09:06:44,824 INFO  namenode.FileJournalManager
    (FileJournalManager.java:finalizeLogSegment(130)) - Finalizing edits
    file
    /ebs/hadoop/hdfs/namenode/current/edits_inprogress_0000000000000842249
    -> /ebs/hadoop/hdfs/name node/current/edits_0000000000000842249-0000000000000842250 2014-07-20
    09:06:44,874 INFO  blockmanagement.CacheReplicationMonitor
    (CacheReplicationMonitor.java:run(168)) - Shutting down
    CacheReplicationMonitor 2014-07-20 09:06:44,876 INFO 
    namenode.FSNamesystem (FSNamesystem.java:startStandbyServices(1136)) -
    Starting services required for standby state 2014-07-20 09:06:44,927
    INFO  ha.EditLogTailer (EditLogTailer.java:(117)) - Will roll
    logs on active node at hadoop-client-us-west-1b/10.0.254.10:8020 every
    120 seconds. 2014-07-20 09:06:44,929 INFO  ha.StandbyCheckpointer
    (StandbyCheckpointer.java:start(129)) - Starting standby checkpoint
    thread... Checkpointing active NN at
    http:// hadoop-client-us-west-1b:50070 Serving checkpoints at
    http:// hadoop-client-us-west-1a:50070 2014-07-20 09:06:44,930 INFO 
    ipc.Server (Server.java:run(2027)) - IPC Server handler 3 on 8020,
    call org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from
    10.0.1.50:57297 Call#8431877 Retry#0: org.apache.hadoop.ipc.StandbyException: Operation category READ is not
    supported in state standby 2014-07-20 09:06:44,930 INFO  ipc.Server
    (Server.java:run(2027)) - IPC Server handler 16 on 8020, call
    org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from
    10.0.1.50:57294 Call#130105071 Retry#0: org.apache.hadoop.ipc.StandbyException: Operation category READ is not
    supported in state standby 2014-07-20 09:06:44,940 INFO  ipc.Server
    (Server.java:run(2027)) - IPC Server handler 14 on 8020, call
    org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from
    10.0.1.50:57294 Call#130105072 Retry#0: org.apache.hadoop.ipc.StandbyException: Operation category READ is not
    supported in state standby</p>
  </blockquote>
  
  <p><strong>Edit:13th August 2014</strong> 
  We were able to found out root cause of namenode switching, namenode getting lots of file info requests and then namenode switching was happened.</p>
  
  <p>But still could not get resolve <strong>Operation category READ is not supported in state standby</strong> error.</p>
  
  <p><strong>Edit:7th December 2014</strong> 
  We were found that, as the <strong>solution</strong> application need to manually connect with current active namenode once previously active namenode failed. Traffic for namenodes in HA mode are not automatically directed to active node. </p>