How do I recover from Mariadb(10.4.18) binlog corruption I have a 3 node Mariadb Galera cluster running Mariadb 10.4.18 on ubuntu 18.4.
I applied updates to one of the nodes this morning and now Mariadb doesn't start on that node with the following error.
[ERROR] I/O error reading the header from the binary log, errno=175, io cache code=0
[ERROR] I/O error reading the header from the binary log
[ERROR] Can't init tc log
[ERROR] Aborting'
2 or the 3 nodes are running and are up to date.
There are 5 items in the mariadb-bin.index file and all point to files that are 0 length.
I have searched the internet to find out how to recover from this but the advices seems to be rather limited. I have found the following:
https://www.webmaster2020.com/forum/viewtopic.php?t=2716
http://raafat.tawasol.net/mysql-crashes-io-error/
These suggest deleting the corrupt fail and restarting. Is this safe and the best way to recover from this?
Once it became clear that the cluster had been working properly right until the reboot, I restored the node to the last backup and let Galera bring the node back up to date.
In the end it was came back very quickly.