Mysql Cluster Exceeding MaxBufferedEpochs I have an mysql cluster which have 4 api nodes, 2 management nodes and 4 data nodes. Today, I was having problems while trying to connect database and all queries was hanging at "Opening tables" state. After inspecting logs i have found these erros on logs:

Api Node Erros:

2015-08-20 19:44:14 15540 [Note] NDB Schema dist: Data node: 5 failed, subscriber bitmask 00
2015-08-20 19:44:14 15540 [Note] NDB Schema dist: Data node: 6 failed, subscriber bitmask 00
2015-08-20 19:44:14 15540 [Note] NDB Schema dist: Data node: 7 failed, subscriber bitmask 00
2015-08-20 19:44:14 15540 [Note] NDB Schema dist: Data node: 8 failed, subscriber bitmask 00
2015-08-20 19:44:14 15540 [Note] NDB Schema dist: cluster failure at epoch 3313124/17.
2015-08-20 19:44:14 15540 [Note] NDB Binlog: ndb tables initially read only on reconnect.
2015-08-20 19:44:14 15540 [ERROR] /opt/mysql/server-5.6/bin/mysqld: Got temporary error 4028 'Node failure caused abort of transaction' from NDBCLUSTER
2015-08-20 19:44:14 15540 [ERROR] /opt/mysql/server-5.6/bin/mysqld: Sort aborted: Got temporary error 4028 'Node failure caused abort of transaction' from NDBCLUSTER
2015-08-20 19:44:14 15540 [ERROR] Got error 4010 when reading table './database_name/table'
2015-08-20 19:44:14 15540 [Note] NDB Binlog: cluster failure for ./database_name/table_name at epoch 3313124/17.

mysql&gt; show processlists;

Id  User    Host    db  Command Time    State   Info
1   system user     NULL    Daemon  1497    Waiting for ndbcluster to start NULL


Data Node Erros:

2015-08-20 19:44:14 [ndbd] ERROR -- c_gcp_list.seize() failed: gci: 14229759227592721 nodes: 0000000000000000000000000000040000000000000000000000000000001a00
2015-08-20 19:44:14 [ndbd] WARNING -- ACK wo/ gcp record (gci: 3313124/17) ref: 0fa2000b from: 0fa2000b
2015-08-20 19:44:14 [ndbd] WARNING -- ACK wo/ gcp record (gci: 3313124/17) ref: 0fa2000c from: 0fa2000c
2015-08-20 19:44:14 [ndbd] WARNING -- ACK wo/ gcp record (gci: 3313124/17) ref: 0fa2008a from: 0fa2008a


Management Node Erros:

2015-08-20 19:44:14 [MgmtSrvr] INFO     -- Node 5: Disconnecting lagging nodes '0000000000000000000000000000000000000000000000000000000000000200',
2015-08-20 19:44:14 [MgmtSrvr] WARNING  -- Node 5: Disconnecting node 9 because it has exceeded MaxBufferedEpochs (100 &gt; 100), epoch 3313119/4


Detailed logs and configurations

Data node config: 

https://gist.github.com/sdemircan/730fa49fcc14b4376c42


Api node config:

https://gist.github.com/sdemircan/f9d230d32700b86564fd


Management node config:

https://gist.github.com/sdemircan/d6fbd54799daaae01bf2


Api Node logs: 

https://gist.github.com/sdemircan/2d62b1c92176de9de9d3


Data Node logs: 

https://gist.github.com/sdemircan/d0c97b82457a9c33deaa


Data Node logs: 

https://gist.github.com/sdemircan/3faa1e41367bc7655210


Management Node logs:

https://gist.github.com/sdemircan/a026ac57757fafdafaa9


What might bring MaxBufferedEpochs to upper limit?
You may have had a large transaction, a query retrieving many rows, pull back a lot of data and saturate the network connection for node 9. 

On reconnect the API nodes are readonly so it has to be a query executed before this

NDB Binlog: ndb tables initially read only on reconnect