State-of-the-Art Distributed Systems with CoralSequencer

In this article we introduce the big picture of CoralSequencer, a full-fledged, ultra-low-latency, high-reliability, software-based middleware for the development of distributed systems based on asynchronous messages. We discuss CoralSequencer’s main parts and how it uses a sophisticated and low-latency protocol to distribute messages across nodes through reliable UDP multicast.

You should also check the YouTube video below presented by Brian Nigito from Jane Street describing the philosophy behind the CoralSequencer architecture.

Fundamentals

  • What is a distributed system?
    It is an integrated system that spans multiple machines, clients or nodes which execute their tasks in parallel in a non-disruptive way. A distributed system should be robust enough to continue to operate when one or more nodes fail, stop working, lag or are taken down for maintenance. Each machine can run multiple nodes, not just one. Each node communicates with each other by sending messages to the queue (i.e. broker). Nodes can produce and consume messages.
  • What is a Messaging Queue or MQ?
    It is a message-oriented middleware that enables the development of a distributed system by using asynchronous messages for inter-node communication. A traditional approach for message distribution is the publish-subscribe pattern. CoralSequencer uses a different all-messages-to-all-nodes approach, through multicast, by implementing a reliable UDP protocol.
  • What are distributed systems built on top of a MQ good at?
    They are great at implementing applications with a lot of moving parts (i.e. nodes) which need real-time monitoring and real-time reaction capabilities. To summarize they provide: parallelism (nodes can truly run in parallel), tight integration (all nodes see the same messages in the same order), decoupling (nodes can evolve independently), failover/redundancy (when a node fails, another one can be running and building state to take over immediately), scalability/load balancing (just add more nodes), elasticity (nodes can lag during activity peaks without affecting the system as a whole) and resiliency (nodes can fail / stop working without taking the whole system down).
  • What is the advantage of CoralSequencer?
    CoralSequencer is a state-of-the-art implementation of a messaging queue for ultra-low-latency distributed systems that require a high degree of reliability. It has all the Coral Blocks’ advantages: high performance, low latency, low variance, zero garbage and simplicity. As you will see in subsequent articles, CoralSequencer is extremely easy to develop, configure, deploy, monitor and maintain.
  • What are some examples of systems that can be built on top of CoralSequencer?
    Electronic exchanges (ECNs), trading platforms, banking systems, automated online advertising systems, defense/military systems, credit card systems or any distributed system with a large number of transactions that requires all the advantages listed above plus ultra low latency and high reliability.

Quick Facts

  • All nodes read all messages in the exact same order, dropping messages they are not interested in
  • All messages are persisted so late-joining nodes can catch up and build the exact same state as other nodes
  • Message broadcasting is done through a reliable UDP protocol: no message is ever lost
  • No single point of failure, from the software down to the hardware infrastructure
  • Each session is automatically archived with all its messages so it can be replayed later for testing, simulation, analysis and auditing
  • High-level, straightforward API makes it easy to write nodes that publish/consume messages
  • As low-latency as it can be through UDP multicast
  • Zero garbage created per message – no gc overhead
  • Automatic node discovery, making it easy to move your nodes across machines
  • Remote administration, backup, logging, TCP bridge and more

Node Example


package com.coralblocks.coralsequencer.node;

import com.coralblocks.coralsequencer.message.Message;
import com.coralblocks.coralsequencer.mq.Node;
import com.coralblocks.coralreactor.nio.NioReactor;
import com.coralblocks.coralreactor.util.Configuration;

public class SampleNode extends Node {
	
	private final static int PERIOD = 3000; // 3 seconds...
	
	private long time;

	public SampleNode(NioReactor nio, String name, Configuration config) {
	    super(nio, name, config);
    }
	
	@Override
	protected void handleActivated() {
		// this method is called when the node becomes active
		sendCommand();
	}
	
	@Override
	protected void handleEventTimeout(long now, int period) {
		// this method is triggered by the event timeout you are setting in the handleMessage method
		sendCommand();
	}
	
	private void sendCommand() {
		time = System.nanoTime();
		sendCommand("TIME");
	}

	@Override
    protected void handleMessage(boolean isMine, Message msg) {
		
		if (!isMine || isRewinding()) return; // not interested, drop it
		
		System.out.println("Took " + (System.nanoTime() - time) + " nanoseconds to see my message in the bus");
		
		setEventTimeout(PERIOD); // set a trigger to send the command again after 3 seconds (goes like that forever)
    }
}

For more information about CoralSequencer please contact Coral Blocks!