Few scenarios are more terrifying for a database administrator than a split-brain state. In an Elasticsearch cluster, this occurs when a network partition causes nodes to lose contact with each other, leading to the election of two independent master nodes. The result is catastrophic: two versions of "truth," diverging data, and a manual recovery process that often involves painful data loss. To maintain a healthy cluster, you must understand the mechanics of consensus and how Elasticsearch has evolved its elective processes from version 6.x to the modern 8.x era.
The goal is simple: ensure that only one set of master-eligible nodes can form a quorum at any given time. By the end of this guide, you will know how to configure your cluster to resist partitions and how to verify that your election settings align with distributed systems best practices.
TL;DR — To prevent split-brain in Elasticsearch 6.x and earlier, set discovery.zen.minimum_master_nodes to (N/2) + 1. In Elasticsearch 7.x and later, the cluster handles this automatically via a voting configuration, provided you correctly initialize the cluster using cluster.initial_master_nodes.
The Concept of Split-Brain in Distributed Search
In Elasticsearch, the "Master Node" is responsible for cluster-wide actions: creating/deleting indices, tracking node health, and deciding where shards live. A split-brain happens when a network glitch isolates a subset of nodes. If that subset contains master-eligible nodes and they believe the original master has died, they will elect a new one among themselves. Now you have two masters, both accepting write requests. When the network heals, Elasticsearch cannot easily merge these two different timelines, often leading to the deletion of one set of data to match the other.
Modern versions of Elasticsearch (7.0+) replaced the old Zen Discovery with a new Cluster Coordination layer. This transition moved the responsibility from the user (who had to manually calculate quorums) to the software, which now tracks a "Voting Configuration." Understanding this shift is vital for anyone upgrading legacy systems to Elasticsearch 8.x.
When Split-Brain Occurs: Real-World Scenarios
Split-brain isn't just a theoretical edge case; it happens frequently in unstable network environments. The most common trigger is a "Flapping" network interface or a saturated top-of-rack switch. During my time managing a 50-node cluster on Elasticsearch 6.8, we experienced a split-brain because our dedicated master nodes were under-provisioned. CPU spikes caused the nodes to stop responding to pings (heartbeats), leading the data nodes to assume the masters were gone and initiate an election.
Another high-risk scenario is deploying nodes across multiple Availability Zones (AZs) without a witness or a majority in a single zone. If the fiber link between AZ-1 and AZ-2 fails, and both zones have enough master-eligible nodes to meet the minimum requirements, both zones will promote a local master. This is why you should always deploy an odd number of master-eligible nodes—typically three—spread across three zones if possible, or two zones with a tie-breaker in a third.
Cluster Architecture and Consensus Logic
To prevent this, Elasticsearch uses a consensus algorithm. The logic relies on the "Quorum" principle: no action can be taken unless a majority of master-eligible nodes agree. If you have 3 master-eligible nodes, the majority is 2. If the cluster splits into a group of 1 and a group of 2, only the group of 2 can elect a master. The isolated node will realize it cannot reach a majority and will wait in a "no-master" state.
[ Node A ] ---X--- [ Node B ]
(Master) (Eligible)
| |
[ Node C ] ------------/
(Eligible)
In this partition, {B, C} form a majority (2 of 3).
Node A is isolated and steps down.
The math for the required majority is always floor(N/2) + 1. This ensures that even in a network partition, it is mathematically impossible for two different groups to both have a majority. If you have 4 nodes, the majority is 3. If that cluster splits 2-2, neither side can function, which preserves data integrity at the cost of temporary downtime. This is why odd numbers of master-eligible nodes are the gold standard.
Implementing Prevention: Version-Specific Steps
Step 1: Configuration for Elasticsearch 6.x (Legacy)
In older versions, you must manually manage the discovery.zen.minimum_master_nodes setting. If you leave this as the default (1), any single node that loses network connectivity will think it is the only node left and declare itself master.
# elasticsearch.yml for a 3-node master-eligible setup
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["10.0.0.1", "10.0.0.2", "10.0.0.3"]
You must update this setting dynamically if you add or remove master-eligible nodes from the cluster. Failing to increment this number when scaling up is a common cause of future outages.
Step 2: Configuration for Elasticsearch 7.x and 8.x (Modern)
Modern Elasticsearch eliminates the need for minimum_master_nodes. Instead, the cluster manages its own voting configuration. However, you must define the initial set of masters when the cluster first starts up. This is done via the cluster.initial_master_nodes setting.
# Initial bootstrap on Node 1, 2, and 3
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]
discovery.seed_hosts: ["10.0.0.1", "10.0.0.2", "10.0.0.3"]
cluster.initial_master_nodes in your configuration after the cluster has successfully formed. Once the cluster is bootstrapped, it stores the voting configuration in its cluster state on disk. Leaving the setting in the config file can cause issues during subsequent restarts or cluster re-boots.
Step 3: Handling Shard Allocation
While not strictly "consensus," ensuring that shards are not over-allocated during a partition helps recovery. Use the cluster.routing.allocation.awareness.attributes to ensure that replicas are never stored in the same rack or zone as their primaries.
The CAP Theorem Tradeoff: Consistency vs. Availability
When you configure for split-brain prevention, you are choosing Consistency (CP) over Availability (AP). In a partition, the "minority" side of the cluster will stop accepting writes and likely stop serving reads to avoid returning stale data. In most enterprise search scenarios, this is the correct choice.
| Feature | Low Quorum (Unsafe) | High Quorum (Safe) |
|---|---|---|
| Data Integrity | High risk of divergence | Guaranteed consistency |
| Up-time | Stays up during partitions | May go offline if N/2+1 isn't met |
| Write Safety | Possible data loss on merge | Writes only happen on one side |
| Recovery | Manual, complex intervention | Automatic once network heals |
For systems where 100% uptime is required even during network failure, you would need a cross-cluster replication (CCR) strategy where two independent clusters handle the load, but even then, the individual clusters must be protected against internal split-brain.
Proactive Monitoring and Operational Tips
To ensure your architecture remains robust, you should regularly audit the cluster state. I recommend automating a check against the /_nodes/_all/process and /_cluster/state APIs. Specifically, monitor the "voting configuration" in version 7.x+ clusters to ensure it matches your expected count of master-eligible nodes.
- Use exactly 3 master-eligible nodes for small-to-medium clusters.
- Never use an even number of master-eligible nodes (e.g., 2 or 4).
- In 7.x+, ensure
cluster.initial_master_nodesis used only for the first bootstrap. - Monitor the
discovery-gceordiscovery-ec2plugins if running in the cloud to ensure node discovery is dynamic. - Always link your Elasticsearch version to your documentation; version 8.x behavior is vastly different from 6.x.
For further reading, consult the official Elasticsearch documentation on Discovery. Understanding the underlying Raft-like consensus used in newer versions will give you deeper insight into how the software handles edge cases during node failure.
Frequently Asked Questions
Q. How can I tell if my cluster has split-brain?
A. Check the cluster UUID. If you query two different nodes and they report different cluster_uuid values, or if they both claim to be "Master" but don't see each other in _cat/nodes, you have a split-brain. You will also see "master_not_discovered_exception" on one side of the partition.
Q. Why is three nodes the minimum for high availability?
A. With two nodes, the majority (N/2 + 1) is 2. If one node fails, the remaining node only represents 50% of the vote, which is not a majority. Therefore, a 2-node cluster cannot survive the loss of a single node while maintaining strict consensus. Three nodes allow one failure while maintaining a majority of two.
Q. Can I use a single node for master and data?
A. Yes, for development or small workloads. In this case, minimum_master_nodes (or the voting config) is 1. Since there are no other nodes to form a second group, split-brain is impossible. However, this configuration provides zero high availability.
Post a Comment