High-Availability Cluster Design: The 3-Node Quorum

Ask anyone who has designed a resilient system why high-availability clusters so often come in threes, and you will usually get a confident answer that turns out to be only half right. Three nodes is indeed the baseline for genuine high availability, but the reason is not simply that more copies are safer. It comes down to a deceptively subtle problem: how a group of machines agrees on the truth when some of them can no longer talk to each other. The answer to that problem is quorum, and quorum is why two nodes are dangerous and three are the practical minimum.

This article explains the logic from first principles, why naive redundancy fails, what split-brain is and why it is so destructive, how odd-numbered quorum solves it, and how these ideas show up in the real clustered systems that run a private cloud.

Redundancy is not the same as availability

It is easy to assume that adding a second server automatically buys you high availability. If one fails, the other takes over, so you are covered. For a simple active-passive failover this intuition is roughly right, but it hides a dangerous edge case that appears the moment the two nodes are both alive yet unable to communicate.

High availability is not just about surviving a node that dies. A dead node is the easy case, it stops responding, and everyone agrees it is gone. The hard case is the node that is still running but isolated, cut off by a failed network link while it continues to believe it is healthy and in charge. Designing for that scenario, not just for clean failures, is what separates a real HA cluster from a pair of servers that happen to back each other up.

The two-node trap and split-brain

Picture a classic two-node cluster: one active, one standby, sharing responsibility for a database or a virtual IP. The network link between them fails, but both nodes are perfectly healthy. Now each node looks across the link, sees nothing, and reaches the same logical conclusion: my partner has died, so I must take over.

The result is catastrophic. Both nodes now believe they are the sole survivor and the rightful primary. Both mount the shared storage, both accept writes, both answer client requests. This is split-brain, and it is far worse than an outage. An outage means downtime; split-brain means two divergent copies of your data accumulating conflicting changes in parallel. When the link recovers, there is no clean way to reconcile them. You are left choosing which set of writes to discard, which in a transactional system can mean lost orders, corrupted records, or worse.

The root cause is that with two nodes there is no way to distinguish my partner is down from I am the one who is cut off. Each node has exactly one vote, and one vote cannot break a tie. Symmetry is the enemy of agreement.

Quorum: agreeing on a single source of truth

Quorum is the mechanism that breaks the symmetry. The principle is simple: the cluster may only operate if a strict majority of its members can agree they are part of the active group. A node, or a group of nodes, must hold more than half the total votes before it is allowed to act as primary. A minority, no matter how healthy, must step back and refuse to serve.

Apply this to a three-node cluster. Total votes: three. Majority needed: two. Now imagine a network partition that splits the cluster, isolating one node on one side and leaving two together on the other. The pair has two votes, a majority, so it retains quorum and keeps running. The lone node has only one vote, knows it is in the minority, and voluntarily stops serving rather than risk diverging. Split-brain becomes impossible, because both sides cannot simultaneously hold a majority of three.

That is the entire trick. Quorum does not prevent partitions; it guarantees that at most one side of any partition is ever allowed to act, which is exactly what you need to protect data integrity.

Why three, and why odd numbers

This is where the question why not two answers itself. Two nodes cannot form a majority of two without unanimity, and unanimity is precisely what a network partition destroys. With two votes, a one-one split has no majority, so either the cluster halts entirely (no availability) or each side proceeds alone (split-brain). There is no safe outcome. Three is the smallest cluster where a single failure still leaves a functioning majority.

The case against even numbers

If three is good, surely four is better? Counterintuitively, no, and the reason is illuminating. With four nodes you can tolerate one failure (three remain, a majority) but a clean two-two network split leaves neither side with a majority, so the whole cluster freezes. You have added a node and gained no extra failure tolerance over three, while introducing a new tie scenario. Five nodes tolerate two failures and still avoid ties; four tolerate only one, the same as three. This is why HA clusters are almost always built in odd numbers: three, five, seven. Each odd step buys real, additional fault tolerance, while even numbers just add cost and a tie risk.

The quorum-only tie-breaker

Sometimes running three full nodes is overkill, for example when two beefy servers carry the workload comfortably. The elegant compromise is a witness or quorum device: a lightweight third voter that holds no data and runs no workload, existing purely to cast the deciding vote in a partition. Two data nodes plus one tiny witness gives you the three-vote, split-brain-proof topology at a fraction of the cost of a third full server.

Fencing: making sure the loser stays down

Quorum decides who should be primary, but a robust cluster also needs to guarantee that the node which lost the vote actually stops. A minority node that is malfunctioning might not cleanly stand down on its own. This is where fencing (also called STONITH, Shoot The Other Node In The Head) comes in: the cluster forcibly isolates or powers off the losing node, often by cutting its power or network at the hardware level, so there is zero chance it keeps writing to shared storage.

Quorum and fencing are complementary. Quorum is the decision; fencing is the enforcement. Together they ensure not only that the cluster picks a single primary, but that the rejected node cannot cause damage while it is confused about its own status.

How this shows up across a private cloud

These principles are not abstract; they are baked into nearly every clustered system you rely on. Distributed databases like the Galera-based MySQL/MariaDB clusters behind many OpenStack control planes require a majority of nodes online to accept writes, which is exactly why a three-node database cluster is standard. The OpenStack control plane itself, the API services, message queue, and coordination layer, is deployed across three controllers so the loss of any one leaves a functioning majority. Ceph, the distributed storage layer, runs its monitors as an odd-numbered quorum for precisely the same reason: the cluster map must have a single agreed version at all times.

The pattern repeats because the underlying problem is universal. Any system that must stay consistent across multiple machines faces the partition question, and quorum across an odd number of members is the proven answer. Networking redundancy complements this at a different layer, keeping the links between those nodes alive so partitions stay rare in the first place, which is the focus of our guide on MLAG and network redundancy.

Designing for real resilience

A few practical principles follow from all this. Always use an odd number of voting members, three at minimum, five for higher fault tolerance. Spread those members across independent failure domains, separate hosts, separate racks, separate power feeds, so a single physical event cannot take down a majority at once. Configure fencing so a confused minority node is decisively removed. And remember that quorum protects consistency, not capacity: you still need enough surviving nodes to carry the workload after a failure, not merely enough to hold a vote.

This is the architecture clouditiv builds by default. Our OpenStack private clouds deploy across three controller nodes with quorum-based databases, an odd-numbered Ceph monitor cluster, and redundant networking, so a single node or link failure is a non-event rather than an outage, in service of the 99.9 percent uptime our customers depend on. Because the whole cluster is provisioned automatically, that resilient three-node foundation comes online in under an hour rather than taking days of careful manual assembly. You can see how the pieces fit together on our on-premise cloud platform.

The takeaway

Three nodes is the baseline for high availability not because three copies are inherently safer, but because three votes are the smallest number that can form a majority after a failure and so prevent split-brain. Quorum lets a cluster agree on a single source of truth even when the network tears it in two; odd numbers ensure there is always a clear majority; and fencing guarantees the loser stays down. Understand those three ideas and the design choices behind every serious clustered system, from databases to storage to the cloud control plane, stop looking arbitrary and start looking inevitable.