In this article we introduce the big picture of CoralSequencer, a full-fledged, ultra-low-latency, high-reliability, software-based middleware for the development of distributed systems based on asynchronous messages. We discuss CoralSequencer’s main parts and how it uses a sophisticated and low-latency protocol to distribute messages across nodes through reliable UDP multicast.

You should also check the YouTube video below where we present the main characteristics of the sequencer architecture together with some advanced features of CoralSequencer. Note: Contrary to the YouTube video, the video below has no ads.

Quick Facts

• All nodes read all messages in the exact same order, dropping messages they are not interested in
• All messages are persisted so late-joining nodes can rewind and catch up to build the exact same state as other nodes
• Message broadcasting is done through a reliable multicast UDP protocol: no message is ever lost
• Supports all cloud environments through TCP without using multicast.
• Change the transport protocol from UDP to TCP by simply flipping a configuration flag with no code changes.
• Hybrid approach (UDP + TCP) to support extensions of the distributed system in the cloud.
• No single point of failure, from the software down to the hardware infrastructure
• Each session is automatically archived with all its messages so it can be replayed later for testing, simulation, analysis and auditing
• High-level, straightforward API makes it easy to write nodes that publish/consume messages
• As low-latency as it can be through UDP multicast
• Zero garbage created per message – no gc overhead

Quick Features

• Message agnostic – send and receive anything you want
• Automatic replayer discovery through multicast, making it easy to move your replayers across machines
• Message fragmentation at the protocol level, so you can transparently send messages of any size
• Comprehensive test framework for deterministic single-threaded memory-transport automated tests
• Choose your transport protocol through configuration without changing a single line of your application code: TCP (for the cloud), UDP (multicast), Shared-Memory (same machine) and Memory (for tests)
• TCP Rewind
• Non-rewinding nodes
• Transparent batching and in-flight messages
• Shared-memory Dispatcher node to avoid multicast fan-out in machines running several nodes pinned to the same CPU core
• Full CLOUD support through TCP transport. Later to switch from TCP to multicast UDP you can simply flip a configuration flag
• Easy replay a full past session archive file through an offline node for testing, validation, auditing, reports, etc. (from a local file or from a centralized remote server)
• Tiered replayer architecture for scalability (optional)
• Comprehensive, zero-garbage, binary and high-performance native serialization protocol with repeating groups, optional fields, IDL, etc. (optional)
• Full duplex bridges (UDP and TCP)
• Long distance bridges with TCP and UDP redundant channels for performance
• A variety of internal messages providing features like node active/passive, node heartbeats, force passive, etc
• Fully deterministic sequencer clock for the centralized distributed system time
• Local and centralized timers with nanosecond precision
• Sequencer-generated messages
• Hot-Hot nodes in a perfect cluster, using the same node account with different instance IDs
• Multiple sequencers in parallel with cross-connect nodes
• ⁠⁠C++ Node support (write a node in C++ that receives and sends messages through JNI)
• ⁠Nodes can choose from which sequence number to rewind from (allowing for managing state in customer snapshot servers)
• ⁠Nodes can commit a sequence number so that they don’t need to reprocess the whole event-stream in case of rewinding
• Remote administration (telnet, rest and http)
• Logger node
• Archiver node
• Admin Node
• And many others

Node Example

package com.coralblocks.coralsequencer.node;

import java.nio.ByteBuffer;

import com.coralblocks.coralbits.util.ByteBufferUtils;
import com.coralblocks.coralbits.util.DateTimeUtils;
import com.coralblocks.coralreactor.admin.AdminAction;
import com.coralblocks.coralreactor.nio.NioReactor;
import com.coralblocks.coralreactor.util.Configuration;
import com.coralblocks.coralsequencer.message.Message;
import com.coralblocks.coralsequencer.mq.Node;

public class SampleNode extends Node {
	
	public SampleNode(NioReactor nio, String name, Configuration config) {
		
	    super(nio, name, config);
	    
	    addAdminAction(new AdminAction("sendTime") {
			@Override
			public boolean execute(CharSequence args, StringBuilder results) {
				sendTime();
				results.append("Time successfully sent!");
				return true;
			}
	    });
    }
	
	private void sendTime() {
		sendCommand("TIME-" + System.currentTimeMillis());
	}
	
	@Override
    protected void handleMessage(boolean isMine, Message msg) {
		
		if (!isMine) return; // not interested, quickly drop it...
		
		ByteBuffer data = msg.getData(); // the raw bytes of the message...
		
		long epochInNanos = eventStreamEpoch(); // deterministic centralized sequencer clock...
		
		CharSequence now = DateTimeUtils.formatDateTimeInNanos(epochInNanos);
		
		System.out.println("Saw my message at " + now + ": " + ByteBufferUtils.parseString(data));
    }
}

The machine used for the latency benchmarks below was a fast Intel 13th Generation Core i9-13900KS (8 x 3.20GHz Base / 6.00GHz Turbo) Ubuntu box.

NOTE: Everyone’s network environment is different, and we usually have a hard time comparing over-the-wire benchmark numbers. To make this simple we present loopback numbers (i.e. client and server running on the same physical machine but different JVMs) which are easy to compare and weed out external factors, isolating the performance of the application + network code. To calculate total numbers you should add your typical over-the-wire network latency. A 256-byte packet traveling through a 10 Gigabits ethernet will take at least 382 nanoseconds to go from NIC to NIC (ignoring the switch hop). If your ethernet is 1 Gigabits then the latency is at least 3.82 micros on top of CoralSequencer numbers. Another factor is the network card latency. Going from JVM to kernel to NIC can be costly and some good network cards optimize that by offering kernel bypass (i.e. Open OnLoad from SolarFlare).

Latency Numbers

Message Size: 1024 bytes
Messages: 1,000,000
Avg Time: 3.379 micros (round-trip network time)
Min Time: 2.717 micros
Max Time: 73.174 micros
75% = [avg: 3.236 micros, max: 3.428 micros]
90% = [avg: 3.286 micros, max: 3.775 micros]
99% = [avg: 3.359 micros, max: 5.155 micros]
99.9% = [avg: 3.376 micros, max: 5.547 micros]
99.99% = [avg: 3.378 micros, max: 5.719 micros]
99.999% = [avg: 3.378 micros, max: 11.311 micros]

Throughput Numbers

The machine used for the throughput benchmarks below was a fast Intel Xeon E-2288G octa-core (8 x 3.70GHz) Ubuntu box not overclocked.

The throughput numbers below are measured when three nodes are pushing messages to the sequencer as fast as they can. How many messages can the sequencer receive and send out?

Message Size: 256 bytes
Messages Sent: 3,000,000
Total Time: 2.688 secs
Messages per second: 1,116,166
Average Time per message: 895 nanos

The three nodes used to stress out the sequencer had throughput numbers close to:

Message Size: 256 bytes
Messages Sent: 2,000,000
Messages per second: 470,483
Average Time per message: 2.125 micros

NOTE ABOUT BATCHING: The throughput numbers above were calculated with the nodes aggressively batching, in other words, the nodes were aggressively sending more than one message inside the same UDP packet. If you assume that the MTU (i.e. max UDP transmission unit size) is around 1500 bytes, then you can fit five 256-byte messages inside the same UDP packet, increasing throughput considerably.

If you decrease the message size, say to 64 bytes, then you will be able to batch even more, increasing the throughput numbers even further. Below the sequencer throughput numbers when the nodes are aggressively batching 64-byte messages inside the same UDP packet:

Message Size: 64 bytes
Messages Sent: 3,000,000
Total Time: 1.287 secs
Messages per second: 2,330,160
Average Time per message: 429 nanos

The three nodes used to stress out the sequencer had throughput numbers close to:

Message Size: 64 bytes
Messages Sent: 2,000,000
Messages per second: 1,641,239
Average Time per message: 609 nanos

Below we present the sequencer throughput numbers when no batching at the node level is taking place, in other words, when the nodes are sending only one 256-byte message per UDP packet.

Message Size: 256 bytes
Messages Sent: 2,000,000
Total Time: 2.543 secs
Messages per second: 786,497
Average Time per message: 1.271 micros

The three nodes used to stress out the sequencer had throughput numbers close to:

Message Size: 256 bytes
Messages Sent: 2,000,000
Messages per second: 275,861
Average Time per message: 3.625 micros

The worst case scenario is when we have a big message filling up the entire UDP packet. Note that with this big message size not even the sequencer will be able to batch anything. Below the throughput numbers when the nodes are sending a 1400-byte message.

Message Size: 1,400 bytes
Messages Sent: 2,000,000
Total Time: 6.097 secs
Messages per second: 328,025
Average Time per message: 3.048 micros

The three nodes used to stress out the sequencer had throughput numbers close to:

Message Size: 1,400 bytes
Messages Sent: 2,000,000
Messages per second: 111,119
Average Time per message: 8.999 micros

Conclusion

CoralSequencer can sustain a throughput of 2 million messages per second if batching is used. The round-trip latencies are close to 4.7 micros per message (256 bytes). Without batching at the node level, the sequencer throughput number is around 780 thousand messages per second for a 256-byte message. For the worst case scenario, a 1400-byte message inside the UDP packet, the sequencer throughput number is around 320 thousand messages per second.

package com.coralblocks.coralsequencer.node;

import java.nio.ByteBuffer;

import com.coralblocks.coralbits.ts.TimeUnit;
import com.coralblocks.coralbits.util.ByteBufferUtils;
import com.coralblocks.coralreactor.nio.NioReactor;
import com.coralblocks.coralreactor.util.Configuration;
import com.coralblocks.coralsequencer.message.Message;
import com.coralblocks.coralsequencer.mq.Node;

public class SampleNode extends Node {
	
	private final static int PERIOD = 3; // 3 seconds...
	
	public SampleNode(NioReactor nio, String name, Configuration config) {
	    super(nio, name, config);
    }
	
	@Override
	protected void handleActivated() {
		// this method is called when the node becomes active
		sendCommand();
	}
	
	@Override
	protected void handleDeactivated() {
		// called when a node has been deactivated
		// once deactivated a node will not send commands
		removeEventTimeout(); // turn off event timeout if set
	}
	
	@Override
	protected void handleEventTimeout(long now, long period, TimeUnit unit) {
		// this method is triggered by the event timeout you are setting in the handleMessage method
		// Note: it is triggered only once so you must re-register the timeout if you want to do it again (it is not a loop timer)
		sendCommand();
	}
	
	private void sendCommand() {
		sendCommand("TIME-" + System.currentTimeMillis());
	}

	@Override
    protected void handleMessage(boolean isMine, Message msg) {
		
		if (!isMine || isRewinding()) return; // not interested, quickly ignore them...
		
		ByteBuffer data = msg.getData();
		
		System.out.println("Saw my message in the event-stream: " + ByteBufferUtils.parseString(data));
		
		setEventTimeout(PERIOD, TimeUnit.SECONDS); // set a trigger to send the command again after 3 seconds
    }
}

Note that every command sent to the sequencer by a node will make the sequencer send a corresponding message to the event-stream with the sender’s account. That’s how the node makes sure that its command was received and processed by the sequencer. You don’t need to worry about that or do anything, but under the hood CoralSequencer will resend the command if it does not see the corresponding message (i.e. the ack) in the event-stream after N milliseconds. Again, this is totally transparent for the developer coding the node, as you can see in the source code above.

A node can be read-only, in other words, it will only listen to the event-stream and never send any command to the sequencer. Our node above does send a command to the sequencer (i.e. it is not read-only) using the sendCommand(String) method. Besides that method, you can also use sendCommand(byte[]), sendCommand(ByteBuffer) and sendCommand(Proto).

The isMine flag passed by the handleMessage(boolean, Message) method is important as it tells you if you are receiving a message that belongs to this node or not. Recall that the sequencer always broadcasts all messages to all nodes, so you will be seeing messages from other nodes in this method. If you are only interested in your messages you can quickly drop them by checking the isMine boolean.

Another important check is done with isRewinding(). The first time the node connects to the sequencer it will receive a replay of all the previous messages from the current session, in a process called rewinding. You can use this messages to rebuild state if you need to. In our simple example we don’t want to do anything with past messages so we simply drop them.

Configuring the Node

Below the CoralSequencer DSL to configure a node:

# allow this node to be managed by telnet
VM addAdmin telnet 51

# creates the node (account = NODE1)
VM newNode NODE1 com.coralblocks.coralsequencer.node.SampleNode

# the lines below can also be executed manually by admin
NODE1 open
NODE1 activate

Add the lines above to a file time.mq and use the script ./bin/start.sh to execute the DSL and start the node:

Managing the Node

Because we configured the telnet admin in the DSL above, we can telnet to the admin port (i.e. 50000 + port id) to execute DSL commands on the node. For example, the last lines to open and activate the node can be executed manually through telnet, as the example below shows:

NOTE: The highlighted lines below are the commands executed

$ telnet localhost 50051
Trying ::1...
Connected to localhost (::1).
Escape character is '^]'.

Hi! What can I do for you? You can start by typing 'list'...

list NODE1

NODE1 open
NODE1 close
NODE1 setMessageReceiver
NODE1 setCommandSender
NODE1 activate
NODE1 sendCommand
NODE1 status

NODE1-CommandSender-255.255.255.255:60010
NODE1-MessageReceiver-0.0.0.0:60066

NODE1 open

NODE1 was opened!

NODE1 activate true

activate called!

Conclusion

Writing a Node using CoralSequencer is extremely easy. Moreover you can configure and manage your nodes using CoralSequencer’s DSL.