Repository navigation
Cluster: a member learnt just before a whole-cluster restart is forgotten and can't rejoin without a new MEET #808
Description
Found while fixing a flaky test (#807).
What happens: nodes save nodes.conf on a background thread shortly after a topology change, and other masters learn of a new replica by gossip up to one heartbeat (1 s) later. If every master is stopped (or crashes) before any of them has saved the new replica, they restart without it. The replica still knows the masters and keeps sending heartbeats, but a node that isn't a member can't join by heartbeat (by design, as in Redis: it needs a CLUSTER MEET), so the masters never learn it again. The cluster stays at cluster_known_nodes:3; the replica keeps replicating from its master, but it is invisible to CLUSTER NODES/CLUSTER SHARDS on the masters and can't be chosen in a failover.
No data or acknowledged writes are lost. The window is about a second after a node joins, and needs all masters to stop inside it.
Ideas:
- Save
nodes.confbefore acknowledging the change that added the node (CLUSTER MEET/CLUSTER REPLICATEreply, or the first gossip that adds a member), as Redis does before it replies. - Or let a node that restarts with fewer members than a peer claims re-handshake with nodes that name it as their master in their heartbeats.
- A test: add a replica, stop all masters within 100 ms, restart, and expect every node to list four nodes.
Activity