The CoralSequencer architecture uses manual failover for the primary sequencer by design. That choice is not an omission and it is not a limitation. It is a deliberate trade-off. It keeps the normal message path as fast, simple, and deterministic as possible, and handle the rare leader failure with an informed operational decision, rather than with something that adds coordination cost to every message. In this article we’ll explain why manual failover can be preferable to automatic leader election for the high-performance primary sequencer.
The Normal Path Must Stay Fast
In an automatic leader election architecture, the leader normally has to replicate each log entry to a majority quorum before the entry can be considered committed. This is the correct design for that kind of distributed consensus system, but it also places a costly coordination procedure on the critical path.
For a low-latency sequencer, that cost matters. The sequencer should be able to assign the next sequence number and immediately distribute the message to all nodes. It should not have to first send the message to backup leaders and wait for acknowledgements from a majority before the message can move forward.
By using manual failover, CoralSequencer keeps the backups out of the hot path. They still receive the sequenced messages, maintain their current position, and are ready to be activated if necessary, but they do not add a quorum round trip to every message. The result is lower latency, higher throughput, and a simpler runtime model.
Financial Systems Require Human Supervision
Regardless of whether a sequencer failover is ever needed, financial systems involving money and risk should not operate without human supervision. In this environment, a failover is not merely a technical event. It can affect orders, executions, positions, risk limits, market connectivity, client flows, and downstream systems.
For that reason, failover should not happen automatically just because a timeout was triggered. The nature of the business requires judgment, verification, and accountability. There should always be an operator monitoring the system, watching the logs, understanding what is happening, and deciding when it is safe and appropriate to activate a backup sequencer.
The Operator Has Better Context
When a leader appears to be having problems, the right action depends on the situation. Is the leader process down? Is the machine unreachable? Is there packet loss? Is a network link unstable? Are all backups equally up to date? These are operational questions, and the aggregated logs in front of the operator usually contain the answer.
A human operator can look at the logs, the network status, the machine state, and the current sequence numbers reported by the backups. With that information, the operator can better choose the best backup to activate, instead of blindly accepting the first node that wins an automatic election.
Automatic election algorithms are very useful when the system must continue without human involvement. But they do not understand the whole operational picture. They react to timeouts and protocol rules. A human can assess the cause of the failure and make a more deliberate decision.
Manual Failover Helps Avoid Split Brain
One of the most important failover rules is simple: do not activate a new leader while the old leader might still be alive and able to send messages. If two leaders are active at the same time, the system can split into two competing timelines.
Manual failover gives the operator a chance to verify that the problematic leader is really dead, offline, isolated, or really unable to continue. The operator can close it, kill its process, power off its machine if necessary and confirm that the failing primary sequencer has really gone offline before promoting a backup sequencer.
This extra verification is valuable because failover is not only about restoring service. It is also about preserving a single authoritative sequence of messages.
Leader Failure Is Rare
In this architecture, a leader failover is expected to happen because of hardware or infrastructure failure, not because of bugs or crashes in the sequencer software. The sequencer is deterministic and all its backup instances are running its exact same code.
If there is a software bug that crashes the primary, the same bug will also crash the backups, because they are processing the same deterministic message flow. An automatic election would not solve that kind of failure. It would only move leadership to another instance running the same code and likely hitting the same problem, if there is any backup instance left at all.
For that reason, it does not make sense to pay a continuous performance and complexity cost for something that should happen rarely. The system should optimize for the common case: the leader is healthy, messages are flowing, and every node is receiving the same ordered stream.
No Guessing About The Best Backup
A common concern with failover without backup message acknowledgements is that, during a failover, messages might be missed by some nodes but not by other nodes, creating an inconsistency. The important point is that the operator can easily address that problem too. The backup sequencers (as well as the replayers and archivers) always show their current sequence number in the logs when they time out. Therefore, during a failover procedure, the operator can look at the aggregated logs to activate the backup sequencer with the highest sequence number. That backup is the one that has seen the most recent sequenced message. This makes the failover decision explicit and observable.
Instead of hiding the decision inside an election protocol, the system exposes the information needed so that the operator can make the safest promotion. The latest sequence number, observable and confirmed in the aggregated logs, becomes the evidence for choosing the new leader.
Conclusion
Manual failover is a deliberate part of the sequencer architecture. It has been successfully used by financial companies for almost 3 decades now. It removes quorum acknowledgements from the normal message path, keeps latency low, avoids unnecessary complexity, and lets an operator make an informed decision when a (very) rare primary sequencer failure actually happens.
For systems where automatic recovery is more important than raw performance, a consensus algorithm can be a good choice. But for a high-performance sequencer, the priority is different: keep the leader fast, avoid complexity in the hot path, keep ordering deterministic, and use the aggregated logs and sequence numbers in order to conscientiously promote the right backup sequencer for the rare occasions where failover is truly needed.