FAST support with CoralFIX

Some exchanges (like CME and BMF/Bovespa) use the FAST protocol for market data streaming. The FAST protocol turns FIX messages into compressed binary and vice-versa saving a lot of bandwidth. In this article we show how CoralFIX takes care of all FAST complexity and translates FAST bits into a FIX message at great speed, without producing any garbage and with auto-generated JIT-friendly source code to ensure maximum performance and low variance. In a following article we will present our FAST decoder benchmarks.

The FAST Template

The exchanges provide their FAST templates in XML format so that you can know how to decode their market data FAST stream. For an example of a XML template, you can click here for the BMF/Bovespa templates. Below we list the template for the FIX message MDIncRefresh_126:


<template xmlns="http://www.fixprotocol.org/ns/fast/td/1.1" name="MDIncRefresh_126" id="126" dictionary="126">
   <!-- desc="PREVIOUS VERSION WAS 123" -->
   <string name="ApplVerID" id="1128">
      <constant value="9" />
   </string>
   <string name="MessageType" id="35">
      <constant value="X" />
   </string>
   <uInt32 name="MsgSeqNum" id="34" />
   <uInt64 name="SendingTime" id="52" />
   <uInt32 name="TradeDate" id="75" />
   <sequence name="MDEntries">
      <length name="NoMDEntries" id="268" />
      <uInt32 name="MDUpdateAction" id="279">
         <copy value="1" />
      </uInt32>
      <uInt32 name="MDPriceLevel" id="1023" presence="optional">
         <increment />
      </uInt32>
      <string name="MDEntryType" id="269">
         <copy value="0" />
      </string>
      <string name="Symbol" id="55" presence="optional" />
      <uInt32 name="SecurityIDSource" id="22">
         <constant value="8" />
      </uInt32>
      <string name="SecurityExchange" id="207">
         <constant value="BVMF" />
      </string>
      <uInt64 name="SecurityID" id="48">
         <copy />
      </uInt64>
      <uInt32 name="RptSeq" id="83">
         <increment />
      </uInt32>
      <string name="QuoteCondition" id="276" presence="optional">
         <default />
      </string>
      <decimal name="MDEntryPx" id="270" presence="optional">
         <exponent>
            <default value="-2" />
         </exponent>
         <mantissa>
            <delta />
         </mantissa>
      </decimal>
      <uInt32 name="NumberOfOrders" id="346" presence="optional">
         <default />
      </uInt32>
      <string name="PriceType" id="423" presence="optional" />
      <uInt32 name="MDEntryTime" id="273">
         <copy />
      </uInt32>
      <int64 name="MDEntrySize" id="271" presence="optional">
         <delta />
      </int64>
      <uInt32 name="MDEntryDate" id="272" presence="optional">
         <copy />
      </uInt32>
      <uInt32 name="MDInsertDate" id="37016" presence="optional">
         <copy />
      </uInt32>
      <uInt32 name="MDInsertTime" id="37017" presence="optional">
         <copy />
      </uInt32>
      <string name="MDStreamID" id="1500" presence="optional">
         <default />
      </string>
      <string name="Currency" id="15" presence="optional">
         <copy />
      </string>
      <string name="TradingSessionID" id="336" presence="optional">
         <default value="2" />
      </string>
      <decimal name="NetChgPrevDay" id="451" presence="optional">
         <exponent>
            <default />
         </exponent>
         <mantissa>
            <delta />
         </mantissa>
      </decimal>
      <uInt32 name="SellerDays" id="287" presence="optional">
         <default />
      </uInt32>
      <uInt64 name="TradeVolume" id="1020" presence="optional">
         <delta />
      </uInt64>
      <string name="TickDirection" id="274" presence="optional">
         <default />
      </string>
      <string name="TradeCondition" id="277" presence="optional" />
      <uInt32 name="OpenCloseSettleFlag" id="286" presence="optional" />
      <string name="OrderID" id="37" presence="optional">
         <default />
      </string>
      <string name="TradeID" id="1003" presence="optional">
         <default />
      </string>
      <string name="MDEntryBuyer" id="288" presence="optional">
         <default />
      </string>
      <string name="MDEntrySeller" id="289" presence="optional">
         <default />
      </string>
      <uInt32 name="MDEntryPositionNo" id="290" presence="optional">
         <default />
      </uInt32>
      <string name="SettlType" id="63" presence="optional">
         <default />
      </string>
      <uInt32 name="SettlDate" id="64" presence="optional">
         <default />
      </uInt32>
      <uInt32 name="SettlePriceType" id="731" presence="optional" />
      <string name="PriceBandType" id="6939" presence="optional">
         <default />
      </string>
      <uInt32 name="PriceLimitType" id="1306" presence="optional">
         <default />
      </uInt32>
      <decimal name="LowLimitPrice" id="1148" presence="optional">
         <exponent>
            <default />
         </exponent>
         <mantissa>
            <delta />
         </mantissa>
      </decimal>
      <decimal name="HighLimitPrice" id="1149" presence="optional">
         <exponent>
            <default />
         </exponent>
         <mantissa>
            <delta />
         </mantissa>
      </decimal>
      <decimal name="TradingRefPrice" id="1150" presence="optional">
         <exponent>
            <default />
         </exponent>
         <mantissa>
            <delta />
         </mantissa>
      </decimal>
   </sequence>
</template>


The Generated Source Code

Fortunately with CoralFIX you don’t have to understand or care about this FAST template or the details of the FAST protocol. All you have to do is provide the URL of the exchange XML to CoralFIX and it will generate a low-latency, garbage-free FAST decoder for you. The template above would generate the following source code:


package com.coralblocks.coralmd.parser.bvmf.derivatives.fast;

import static com.coralblocks.coralfix.fast.TypeDecoder.*;
import java.nio.ByteBuffer;
import com.coralblocks.coralbits.NullableStringBuilder;
import com.coralblocks.coralfix.FixMessage;
import com.coralblocks.coralfix.fast.FastTemplate;
import com.coralblocks.coralfix.fast.PMap;
import com.coralblocks.coralfix.fast.field.*;
import com.coralblocks.coralfix.*;

public final class MDIncRefresh_126 implements FastTemplate {

	@Override
	public int getTemplateId() {
		return 126;
	}

	private final PMap pMap1 = new PMap();

	private final FastField field1 = new Constant(1128, false, "9"); // ApplVerID
	private final FastField field2 = new Constant(35, false, "X"); // MessageType
	private final FastField field3 = new NoOperationUnsignedNumber(34, false); // MsgSeqNum
	private final FastField field4 = new NoOperationUnsignedNumber(52, false); // SendingTime
	private final FastField field5 = new NoOperationUnsignedNumber(75, false); // TradeDate
	private final FastField field6 = new CopyUnsignedNumber(279, false, "1"); // MDUpdateAction
	private final FastField field7 = new IncrementUnsignedNumber(1023, true, null); // MDPriceLevel
	private final FastField field8 = new CopyString(269, false, "0"); // MDEntryType
	private final FastField field9 = new NoOperationString(55, true); // Symbol
	private final FastField field10 = new Constant(22, false, "8"); // SecurityIDSource
	private final FastField field11 = new Constant(207, false, "BVMF"); // SecurityExchange
	private final FastField field12 = new CopyUnsignedNumber(48, false, null); // SecurityID
	private final FastField field13 = new IncrementUnsignedNumber(83, false, null); // RptSeq
	private final FastField field14 = new DefaultString(276, true, null); // QuoteCondition
	private final FastField field15 = new DefaultDeltaDecimal(270, true, -2, 0); // MDEntryPx
	private final FastField field16 = new DefaultUnsignedNumber(346, true, null); // NumberOfOrders
	private final FastField field17 = new NoOperationString(423, true); // PriceType
	private final FastField field18 = new CopyUnsignedNumber(273, false, null); // MDEntryTime
	private final FastField field19 = new DeltaSignedNumber(271, true, null); // MDEntrySize
	private final FastField field20 = new CopyUnsignedNumber(272, true, null); // MDEntryDate
	private final FastField field21 = new CopyUnsignedNumber(37016, true, null); // MDInsertDate
	private final FastField field22 = new CopyUnsignedNumber(37017, true, null); // MDInsertTime
	private final FastField field23 = new DefaultString(1500, true, null); // MDStreamID
	private final FastField field24 = new CopyString(15, true, null); // Currency
	private final FastField field25 = new DefaultString(336, true, "2"); // TradingSessionID
	private final FastField field26 = new DefaultDeltaDecimal(451, true, NULL_VALUE, 0); // NetChgPrevDay
	private final FastField field27 = new DefaultUnsignedNumber(287, true, null); // SellerDays
	private final FastField field28 = new DeltaUnsignedNumber(1020, true, null); // TradeVolume
	private final FastField field29 = new DefaultString(274, true, null); // TickDirection
	private final FastField field30 = new NoOperationString(277, true); // TradeCondition
	private final FastField field31 = new NoOperationUnsignedNumber(286, true); // OpenCloseSettleFlag
	private final FastField field32 = new DefaultString(37, true, null); // OrderID
	private final FastField field33 = new DefaultString(1003, true, null); // TradeID
	private final FastField field34 = new DefaultString(288, true, null); // MDEntryBuyer
	private final FastField field35 = new DefaultString(289, true, null); // MDEntrySeller
	private final FastField field36 = new DefaultUnsignedNumber(290, true, null); // MDEntryPositionNo
	private final FastField field37 = new DefaultString(63, true, null); // SettlType
	private final FastField field38 = new DefaultUnsignedNumber(64, true, null); // SettlDate
	private final FastField field39 = new NoOperationUnsignedNumber(731, true); // SettlePriceType
	private final FastField field40 = new DefaultString(6939, true, null); // PriceBandType
	private final FastField field41 = new DefaultUnsignedNumber(1306, true, null); // PriceLimitType
	private final FastField field42 = new DefaultDeltaDecimal(1148, true, NULL_VALUE, 0); // LowLimitPrice
	private final FastField field43 = new DefaultDeltaDecimal(1149, true, NULL_VALUE, 0); // HighLimitPrice
	private final FastField field44 = new DefaultDeltaDecimal(1150, true, NULL_VALUE, 0); // TradingRefPrice


	@Override
	public final void reset() {
		field1.reset();
		field2.reset();
		field3.reset();
		field4.reset();
		field5.reset();
		field6.reset();
		field7.reset();
		field8.reset();
		field9.reset();
		field10.reset();
		field11.reset();
		field12.reset();
		field13.reset();
		field14.reset();
		field15.reset();
		field16.reset();
		field17.reset();
		field18.reset();
		field19.reset();
		field20.reset();
		field21.reset();
		field22.reset();
		field23.reset();
		field24.reset();
		field25.reset();
		field26.reset();
		field27.reset();
		field28.reset();
		field29.reset();
		field30.reset();
		field31.reset();
		field32.reset();
		field33.reset();
		field34.reset();
		field35.reset();
		field36.reset();
		field37.reset();
		field38.reset();
		field39.reset();
		field40.reset();
		field41.reset();
		field42.reset();
		field43.reset();
		field44.reset();
	}

	@Override
	public final void decode(PMap pMap, ByteBuffer bb, FixMessage fixMsg, boolean[] tagsToInclude) {

		boolean[] tags = fixMsg != null ? tagsToInclude : null;

		field1.decode(bb, pMap, fixMsg, tags); // ApplVerID (1128)
		field2.decode(bb, pMap, fixMsg, tags); // MessageType (35)
		field3.decode(bb, pMap, fixMsg, tags); // MsgSeqNum (34)
		field4.decode(bb, pMap, fixMsg, tags); // SendingTime (52)
		field5.decode(bb, pMap, fixMsg, tags); // TradeDate (75)

		int noMDEntries = (int) readUnsignedNumber(bb, false, false);

		FixGroup noMDEntriesGroup1 = null;

		if (tagsToInclude[268]) noMDEntriesGroup1 = fixMsg.createGroup(268);

		for(int i1 = 0; i1 < noMDEntries; i1++) {

			FixGroupElement elem1 = noMDEntriesGroup1 != null ? noMDEntriesGroup1.nextElement() : null;

			pMap1.read(bb);

			boolean[] tags1 = elem1 != null ? tagsToInclude : null;

			field6.decode(bb, pMap1, elem1, tags1); // MDUpdateAction (279)
			field7.decode(bb, pMap1, elem1, tags1); // MDPriceLevel (1023)
			field8.decode(bb, pMap1, elem1, tags1); // MDEntryType (269)
			field9.decode(bb, pMap1, elem1, tags1); // Symbol (55)
			field10.decode(bb, pMap1, elem1, tags1); // SecurityIDSource (22)
			field11.decode(bb, pMap1, elem1, tags1); // SecurityExchange (207)
			field12.decode(bb, pMap1, elem1, tags1); // SecurityID (48)
			field13.decode(bb, pMap1, elem1, tags1); // RptSeq (83)
			field14.decode(bb, pMap1, elem1, tags1); // QuoteCondition (276)
			field15.decode(bb, pMap1, elem1, tags1); // MDEntryPx (270)
			field16.decode(bb, pMap1, elem1, tags1); // NumberOfOrders (346)
			field17.decode(bb, pMap1, elem1, tags1); // PriceType (423)
			field18.decode(bb, pMap1, elem1, tags1); // MDEntryTime (273)
			field19.decode(bb, pMap1, elem1, tags1); // MDEntrySize (271)
			field20.decode(bb, pMap1, elem1, tags1); // MDEntryDate (272)
			field21.decode(bb, pMap1, elem1, tags1); // MDInsertDate (37016)
			field22.decode(bb, pMap1, elem1, tags1); // MDInsertTime (37017)
			field23.decode(bb, pMap1, elem1, tags1); // MDStreamID (1500)
			field24.decode(bb, pMap1, elem1, tags1); // Currency (15)
			field25.decode(bb, pMap1, elem1, tags1); // TradingSessionID (336)
			field26.decode(bb, pMap1, elem1, tags1); // NetChgPrevDay (451)
			field27.decode(bb, pMap1, elem1, tags1); // SellerDays (287)
			field28.decode(bb, pMap1, elem1, tags1); // TradeVolume (1020)
			field29.decode(bb, pMap1, elem1, tags1); // TickDirection (274)
			field30.decode(bb, pMap1, elem1, tags1); // TradeCondition (277)
			field31.decode(bb, pMap1, elem1, tags1); // OpenCloseSettleFlag (286)
			field32.decode(bb, pMap1, elem1, tags1); // OrderID (37)
			field33.decode(bb, pMap1, elem1, tags1); // TradeID (1003)
			field34.decode(bb, pMap1, elem1, tags1); // MDEntryBuyer (288)
			field35.decode(bb, pMap1, elem1, tags1); // MDEntrySeller (289)
			field36.decode(bb, pMap1, elem1, tags1); // MDEntryPositionNo (290)
			field37.decode(bb, pMap1, elem1, tags1); // SettlType (63)
			field38.decode(bb, pMap1, elem1, tags1); // SettlDate (64)
			field39.decode(bb, pMap1, elem1, tags1); // SettlePriceType (731)
			field40.decode(bb, pMap1, elem1, tags1); // PriceBandType (6939)
			field41.decode(bb, pMap1, elem1, tags1); // PriceLimitType (1306)
			field42.decode(bb, pMap1, elem1, tags1); // LowLimitPrice (1148)
			field43.decode(bb, pMap1, elem1, tags1); // HighLimitPrice (1149)
			field44.decode(bb, pMap1, elem1, tags1); // TradingRefPrice (1150)

		}
	}
}

CoralFIX generates garbage-free and JIT-friendly (Hotspot Just-in-Time compiler) source code to ensure maximum performance and low variance.


Generating the Source Code

To generate source code for all templates from an exchange XML file, you can use the code below. Just run it and a decoder for each message will be generated inside the specified directory and package:

import com.coralblocks.coralfix.fast.TemplatesCodeGenerator;

public class GenerateTemplates {
	
	public static void main(String[] args) throws Throwable {
		
		TemplatesCodeGenerator g = new TemplatesCodeGenerator();
		
		String url = "ftp://ftp.bmf.com.br/FIXFAST/templates/Production/templates-UMDF-NTP.xml";
		
		String outputDir = "./src/main/java/com/coralblocks/coralmd/parser/bvmf/derivatives/fast";
		
		String outputPackage = "com.coralblocks.coralmd.parser.bvmf.derivatives.fast";
		
		String parserName = "BmfFastParser";
		
		g.parseXML(url, outputDir, outputPackage, parserName);
	}
}

Notice that it will even create a parser class com.coralblocks.coralmd.parser.bvmf.derivatives.fast.BmfFastParser for you!

Using the Parser

Once the source code is generated you can start using the parser right away to parse FAST bits into FIX messages:


// instantiate the parser
FastParser fastParser = new BmfFastParser();

// parse the FAST bits from a ByteBuffer
FixMessage fixMsg = fastParser.parse(byteBuffer);


Excluding tags we don’t care

Usually a market data FIX message comes with many FIX tags that we don’t care. To save time during the FAST decoding process, we can specify the tags that we care and the FAST parser will not waste time parsing the other tags. As a result, the parsing will be much faster and the resulting FixMessage object will only have the tags we care about. You specify tags by message type, in other words, by the FAST template id as the example below shows:

	private final FastParser fastParser = new BmfFastParser() {
		
		@Override
        protected boolean[] getTagsToInclude(int templateId) {
			
			if (templateId == 111) {

				// MDSecurityList
				
				boolean[] tagsWeAreInterested = getHeaderTags();
				
				includeTags(tagsWeAreInterested, 393, NoRelatedSym, 893, Symbol, SecurityID);

				return tagsWeAreInterested;
			}
			
			if (templateId == 126 || templateId == 123) {
				
				// MDIncRefresh
				
				boolean[] tagsWeAreInterested = getHeaderTags();
				
				includeTags(tagsWeAreInterested, NoMDEntries, MDEntryType, MDUpdateAction, SecurityID, OrderID, MDEntryPositionNo, MDEntryPx, MDEntrySize, TradeCondition, MDEntryBuyer, MDEntrySeller);
				
				return tagsWeAreInterested;
			}
			
			if (templateId == 120 || templateId == 100 || templateId == 49) {
				
				// News
				
				boolean[] tagsWeAreInterested = getHeaderTags();
				
				includeTags(tagsWeAreInterested, 148, NoRelatedSym, SecurityID);
				
				return tagsWeAreInterested;
			}
			
			return getAllTags();
        }
};

Note that if you don’t want to specify the tags for a template you must return getAllTags() so all tags are included in the parsed fix message.

Conclusion

CoralFIX supports the FAST protocol through an auto-generated, ultra-fast, garbage-free, JIT-friendly FAST decoder. You don’t have to understand or care about the FAST protocol details. All you have to do is let CoralFIX generate from the exchange XML templates some JIT-friendly source for you and use it to translate FAST bits into a fully functional FixMessage object.



Article
Name
FAST support with CoralFIX
Company
Summary
CoralFIX supports the FAST protocol through an auto-generated, ultra-fast, garbage-free, JIT-friendly FAST decoder. You don't have to understand or care about the FAST protocol details. All you have to do is let CoralFIX generate from the exchange XML templates some JIT-friendly source for you and use it to translate FAST bits into a fully functional FixMessage object.