Tweets
(2023-04-06 15:48:59)
(2022-11-27 12:55:38)
(2018-07-05 11:47:12)
(2018-06-06 10:05:48)
(2017-08-20 14:27:45)
(2017-07-02 22:40:00)
(2017-06-25 20:47:31)
(2017-06-25 20:38:53)
(2017-06-20 22:17:45)
(2017-06-17 15:45:26)
Seems to be an interesting gotcha with binlogs when setting up new nodes in a Percona XtraDB cluster
Posted on 2015-06-16 @ 10:52:00
When the new server starts for the first time, it creates the binlog index file and an initial binlog file. It then connects to the clusters, finds it's not initialised and starts an SST operation. So far, so good...
The issue I've been hitting is that some part of the process is messing directly with the binary log index file, resulting in the new server refusing to start because a binlog file has an incorrect magic number ("Binlog has bad magic number. It's not a binary log that can be used by this version of MySQL"). Digging into it, I've found that something (I believe it's the xtrabackup transfer) is trying to recreate the index file based on a filename match - which is not only picking up the actual binary logs, but the index file as well. The result is a binary log index file that refers to itself, which when the server tries to read as a binary log file, causes it to choke.
My short-term solution is to specify an explicit value for "log_bin_index" to a different directory than the binlogs are being stored in - the new node seems to bootstrap itself happily enough now. There's still an index file being created in the "wrong" place, which isn't getting updated by the server, but the cluster seems happy enough for now.
There's still some weirdness - the binlog from the donor isn't being referenced at all (going to make hanging slaves from the cluster "fun"), the mystery index file, why whatever is doing this isn't picking up on the explicit binlog index file name - but until I get chance to build a minimum test case for this it'll have to wait. I have more nodes to add to this cluster to ensure the ability to form a quorum...