Some exchanges (like CME and BMF/Bovespa) use the FAST protocol for market data streaming. The FAST protocol turns FIX messages into compressed binary and vice-versa saving a lot of bandwidth. In this article we show how CoralFIX takes care of all FAST complexity and translates FAST bits into a FIX message at great speed, without producing any garbage and with auto-generated JIT-friendly source code to ensure maximum performance and low variance. In a following article we will present our FAST decoder benchmarks.
The FAST Template
The exchanges provide their FAST templates in XML format so that you can know how to decode their market data FAST stream. For an example of a XML template, you can click here for the BMF/Bovespa templates. Below we list the template for the FIX message MDIncRefresh_126:
<template xmlns="http://www.fixprotocol.org/ns/fast/td/1.1" name="MDIncRefresh_126" id="126" dictionary="126"> <!-- desc="PREVIOUS VERSION WAS 123" --> <string name="ApplVerID" id="1128"> <constant value="9" /> </string> <string name="MessageType" id="35"> <constant value="X" /> </string> <uInt32 name="MsgSeqNum" id="34" /> <uInt64 name="SendingTime" id="52" /> <uInt32 name="TradeDate" id="75" /> <sequence name="MDEntries"> <length name="NoMDEntries" id="268" /> <uInt32 name="MDUpdateAction" id="279"> <copy value="1" /> </uInt32> <uInt32 name="MDPriceLevel" id="1023" presence="optional"> <increment /> </uInt32> <string name="MDEntryType" id="269"> <copy value="0" /> </string> <string name="Symbol" id="55" presence="optional" /> <uInt32 name="SecurityIDSource" id="22"> <constant value="8" /> </uInt32> <string name="SecurityExchange" id="207"> <constant value="BVMF" /> </string> <uInt64 name="SecurityID" id="48"> <copy /> </uInt64> <uInt32 name="RptSeq" id="83"> <increment /> </uInt32> <string name="QuoteCondition" id="276" presence="optional"> <default /> </string> <decimal name="MDEntryPx" id="270" presence="optional"> <exponent> <default value="-2" /> </exponent> <mantissa> <delta /> </mantissa> </decimal> <uInt32 name="NumberOfOrders" id="346" presence="optional"> <default /> </uInt32> <string name="PriceType" id="423" presence="optional" /> <uInt32 name="MDEntryTime" id="273"> <copy /> </uInt32> <int64 name="MDEntrySize" id="271" presence="optional"> <delta /> </int64> <uInt32 name="MDEntryDate" id="272" presence="optional"> <copy /> </uInt32> <uInt32 name="MDInsertDate" id="37016" presence="optional"> <copy /> </uInt32> <uInt32 name="MDInsertTime" id="37017" presence="optional"> <copy /> </uInt32> <string name="MDStreamID" id="1500" presence="optional"> <default /> </string> <string name="Currency" id="15" presence="optional"> <copy /> </string> <string name="TradingSessionID" id="336" presence="optional"> <default value="2" /> </string> <decimal name="NetChgPrevDay" id="451" presence="optional"> <exponent> <default /> </exponent> <mantissa> <delta /> </mantissa> </decimal> <uInt32 name="SellerDays" id="287" presence="optional"> <default /> </uInt32> <uInt64 name="TradeVolume" id="1020" presence="optional"> <delta /> </uInt64> <string name="TickDirection" id="274" presence="optional"> <default /> </string> <string name="TradeCondition" id="277" presence="optional" /> <uInt32 name="OpenCloseSettleFlag" id="286" presence="optional" /> <string name="OrderID" id="37" presence="optional"> <default /> </string> <string name="TradeID" id="1003" presence="optional"> <default /> </string> <string name="MDEntryBuyer" id="288" presence="optional"> <default /> </string> <string name="MDEntrySeller" id="289" presence="optional"> <default /> </string> <uInt32 name="MDEntryPositionNo" id="290" presence="optional"> <default /> </uInt32> <string name="SettlType" id="63" presence="optional"> <default /> </string> <uInt32 name="SettlDate" id="64" presence="optional"> <default /> </uInt32> <uInt32 name="SettlePriceType" id="731" presence="optional" /> <string name="PriceBandType" id="6939" presence="optional"> <default /> </string> <uInt32 name="PriceLimitType" id="1306" presence="optional"> <default /> </uInt32> <decimal name="LowLimitPrice" id="1148" presence="optional"> <exponent> <default /> </exponent> <mantissa> <delta /> </mantissa> </decimal> <decimal name="HighLimitPrice" id="1149" presence="optional"> <exponent> <default /> </exponent> <mantissa> <delta /> </mantissa> </decimal> <decimal name="TradingRefPrice" id="1150" presence="optional"> <exponent> <default /> </exponent> <mantissa> <delta /> </mantissa> </decimal> </sequence> </template>
The Generated Source Code
Fortunately with CoralFIX you don’t have to understand or care about this FAST template or the details of the FAST protocol. All you have to do is provide the URL of the exchange XML to CoralFIX and it will generate a low-latency, garbage-free FAST decoder for you. The template above would generate the following source code:
package com.coralblocks.coralmd.parser.bvmf.derivatives.fast; import static com.coralblocks.coralfix.fast.TypeDecoder.*; import java.nio.ByteBuffer; import com.coralblocks.coralbits.NullableStringBuilder; import com.coralblocks.coralfix.FixMessage; import com.coralblocks.coralfix.fast.FastTemplate; import com.coralblocks.coralfix.fast.PMap; import com.coralblocks.coralfix.fast.field.*; import com.coralblocks.coralfix.*; public final class MDIncRefresh_126 implements FastTemplate { @Override public int getTemplateId() { return 126; } private final PMap pMap1 = new PMap(); private final FastField field1 = new Constant(1128, false, "9"); // ApplVerID private final FastField field2 = new Constant(35, false, "X"); // MessageType private final FastField field3 = new NoOperationUnsignedNumber(34, false); // MsgSeqNum private final FastField field4 = new NoOperationUnsignedNumber(52, false); // SendingTime private final FastField field5 = new NoOperationUnsignedNumber(75, false); // TradeDate private final FastField field6 = new CopyUnsignedNumber(279, false, "1"); // MDUpdateAction private final FastField field7 = new IncrementUnsignedNumber(1023, true, null); // MDPriceLevel private final FastField field8 = new CopyString(269, false, "0"); // MDEntryType private final FastField field9 = new NoOperationString(55, true); // Symbol private final FastField field10 = new Constant(22, false, "8"); // SecurityIDSource private final FastField field11 = new Constant(207, false, "BVMF"); // SecurityExchange private final FastField field12 = new CopyUnsignedNumber(48, false, null); // SecurityID private final FastField field13 = new IncrementUnsignedNumber(83, false, null); // RptSeq private final FastField field14 = new DefaultString(276, true, null); // QuoteCondition private final FastField field15 = new DefaultDeltaDecimal(270, true, -2, 0); // MDEntryPx private final FastField field16 = new DefaultUnsignedNumber(346, true, null); // NumberOfOrders private final FastField field17 = new NoOperationString(423, true); // PriceType private final FastField field18 = new CopyUnsignedNumber(273, false, null); // MDEntryTime private final FastField field19 = new DeltaSignedNumber(271, true, null); // MDEntrySize private final FastField field20 = new CopyUnsignedNumber(272, true, null); // MDEntryDate private final FastField field21 = new CopyUnsignedNumber(37016, true, null); // MDInsertDate private final FastField field22 = new CopyUnsignedNumber(37017, true, null); // MDInsertTime private final FastField field23 = new DefaultString(1500, true, null); // MDStreamID private final FastField field24 = new CopyString(15, true, null); // Currency private final FastField field25 = new DefaultString(336, true, "2"); // TradingSessionID private final FastField field26 = new DefaultDeltaDecimal(451, true, NULL_VALUE, 0); // NetChgPrevDay private final FastField field27 = new DefaultUnsignedNumber(287, true, null); // SellerDays private final FastField field28 = new DeltaUnsignedNumber(1020, true, null); // TradeVolume private final FastField field29 = new DefaultString(274, true, null); // TickDirection private final FastField field30 = new NoOperationString(277, true); // TradeCondition private final FastField field31 = new NoOperationUnsignedNumber(286, true); // OpenCloseSettleFlag private final FastField field32 = new DefaultString(37, true, null); // OrderID private final FastField field33 = new DefaultString(1003, true, null); // TradeID private final FastField field34 = new DefaultString(288, true, null); // MDEntryBuyer private final FastField field35 = new DefaultString(289, true, null); // MDEntrySeller private final FastField field36 = new DefaultUnsignedNumber(290, true, null); // MDEntryPositionNo private final FastField field37 = new DefaultString(63, true, null); // SettlType private final FastField field38 = new DefaultUnsignedNumber(64, true, null); // SettlDate private final FastField field39 = new NoOperationUnsignedNumber(731, true); // SettlePriceType private final FastField field40 = new DefaultString(6939, true, null); // PriceBandType private final FastField field41 = new DefaultUnsignedNumber(1306, true, null); // PriceLimitType private final FastField field42 = new DefaultDeltaDecimal(1148, true, NULL_VALUE, 0); // LowLimitPrice private final FastField field43 = new DefaultDeltaDecimal(1149, true, NULL_VALUE, 0); // HighLimitPrice private final FastField field44 = new DefaultDeltaDecimal(1150, true, NULL_VALUE, 0); // TradingRefPrice @Override public final void reset() { field1.reset(); field2.reset(); field3.reset(); field4.reset(); field5.reset(); field6.reset(); field7.reset(); field8.reset(); field9.reset(); field10.reset(); field11.reset(); field12.reset(); field13.reset(); field14.reset(); field15.reset(); field16.reset(); field17.reset(); field18.reset(); field19.reset(); field20.reset(); field21.reset(); field22.reset(); field23.reset(); field24.reset(); field25.reset(); field26.reset(); field27.reset(); field28.reset(); field29.reset(); field30.reset(); field31.reset(); field32.reset(); field33.reset(); field34.reset(); field35.reset(); field36.reset(); field37.reset(); field38.reset(); field39.reset(); field40.reset(); field41.reset(); field42.reset(); field43.reset(); field44.reset(); } @Override public final void decode(PMap pMap, ByteBuffer bb, FixMessage fixMsg, boolean[] tagsToInclude) { boolean[] tags = fixMsg != null ? tagsToInclude : null; field1.decode(bb, pMap, fixMsg, tags); // ApplVerID (1128) field2.decode(bb, pMap, fixMsg, tags); // MessageType (35) field3.decode(bb, pMap, fixMsg, tags); // MsgSeqNum (34) field4.decode(bb, pMap, fixMsg, tags); // SendingTime (52) field5.decode(bb, pMap, fixMsg, tags); // TradeDate (75) int noMDEntries = (int) readUnsignedNumber(bb, false, false); FixGroup noMDEntriesGroup1 = null; if (tagsToInclude[268]) noMDEntriesGroup1 = fixMsg.createGroup(268); for(int i1 = 0; i1 < noMDEntries; i1++) { FixGroupElement elem1 = noMDEntriesGroup1 != null ? noMDEntriesGroup1.nextElement() : null; pMap1.read(bb); boolean[] tags1 = elem1 != null ? tagsToInclude : null; field6.decode(bb, pMap1, elem1, tags1); // MDUpdateAction (279) field7.decode(bb, pMap1, elem1, tags1); // MDPriceLevel (1023) field8.decode(bb, pMap1, elem1, tags1); // MDEntryType (269) field9.decode(bb, pMap1, elem1, tags1); // Symbol (55) field10.decode(bb, pMap1, elem1, tags1); // SecurityIDSource (22) field11.decode(bb, pMap1, elem1, tags1); // SecurityExchange (207) field12.decode(bb, pMap1, elem1, tags1); // SecurityID (48) field13.decode(bb, pMap1, elem1, tags1); // RptSeq (83) field14.decode(bb, pMap1, elem1, tags1); // QuoteCondition (276) field15.decode(bb, pMap1, elem1, tags1); // MDEntryPx (270) field16.decode(bb, pMap1, elem1, tags1); // NumberOfOrders (346) field17.decode(bb, pMap1, elem1, tags1); // PriceType (423) field18.decode(bb, pMap1, elem1, tags1); // MDEntryTime (273) field19.decode(bb, pMap1, elem1, tags1); // MDEntrySize (271) field20.decode(bb, pMap1, elem1, tags1); // MDEntryDate (272) field21.decode(bb, pMap1, elem1, tags1); // MDInsertDate (37016) field22.decode(bb, pMap1, elem1, tags1); // MDInsertTime (37017) field23.decode(bb, pMap1, elem1, tags1); // MDStreamID (1500) field24.decode(bb, pMap1, elem1, tags1); // Currency (15) field25.decode(bb, pMap1, elem1, tags1); // TradingSessionID (336) field26.decode(bb, pMap1, elem1, tags1); // NetChgPrevDay (451) field27.decode(bb, pMap1, elem1, tags1); // SellerDays (287) field28.decode(bb, pMap1, elem1, tags1); // TradeVolume (1020) field29.decode(bb, pMap1, elem1, tags1); // TickDirection (274) field30.decode(bb, pMap1, elem1, tags1); // TradeCondition (277) field31.decode(bb, pMap1, elem1, tags1); // OpenCloseSettleFlag (286) field32.decode(bb, pMap1, elem1, tags1); // OrderID (37) field33.decode(bb, pMap1, elem1, tags1); // TradeID (1003) field34.decode(bb, pMap1, elem1, tags1); // MDEntryBuyer (288) field35.decode(bb, pMap1, elem1, tags1); // MDEntrySeller (289) field36.decode(bb, pMap1, elem1, tags1); // MDEntryPositionNo (290) field37.decode(bb, pMap1, elem1, tags1); // SettlType (63) field38.decode(bb, pMap1, elem1, tags1); // SettlDate (64) field39.decode(bb, pMap1, elem1, tags1); // SettlePriceType (731) field40.decode(bb, pMap1, elem1, tags1); // PriceBandType (6939) field41.decode(bb, pMap1, elem1, tags1); // PriceLimitType (1306) field42.decode(bb, pMap1, elem1, tags1); // LowLimitPrice (1148) field43.decode(bb, pMap1, elem1, tags1); // HighLimitPrice (1149) field44.decode(bb, pMap1, elem1, tags1); // TradingRefPrice (1150) } } }
CoralFIX generates garbage-free and JIT-friendly (Hotspot Just-in-Time compiler) source code to ensure maximum performance and low variance.
Generating the Source Code
To generate source code for all templates from an exchange XML file, you can use the code below. Just run it and a decoder for each message will be generated inside the specified directory and package:
import com.coralblocks.coralfix.fast.TemplatesCodeGenerator; public class GenerateTemplates { public static void main(String[] args) throws Throwable { TemplatesCodeGenerator g = new TemplatesCodeGenerator(); String url = "ftp://ftp.bmf.com.br/FIXFAST/templates/Production/templates-UMDF-NTP.xml"; String outputDir = "./src/main/java/com/coralblocks/coralmd/parser/bvmf/derivatives/fast"; String outputPackage = "com.coralblocks.coralmd.parser.bvmf.derivatives.fast"; String parserName = "BmfFastParser"; g.parseXML(url, outputDir, outputPackage, parserName); } }
Notice that it will even create a parser class com.coralblocks.coralmd.parser.bvmf.derivatives.fast.BmfFastParser
for you!
Using the Parser
Once the source code is generated you can start using the parser right away to parse FAST bits into FIX messages:
// instantiate the parser FastParser fastParser = new BmfFastParser(); // parse the FAST bits from a ByteBuffer FixMessage fixMsg = fastParser.parse(byteBuffer);
Excluding tags we don’t care
Usually a market data FIX message comes with many FIX tags that we don’t care. To save time during the FAST decoding process, we can specify the tags that we care and the FAST parser will not waste time parsing the other tags. As a result, the parsing will be much faster and the resulting FixMessage
object will only have the tags we care about. You specify tags by message type, in other words, by the FAST template id as the example below shows:
private final FastParser fastParser = new BmfFastParser() { @Override protected boolean[] getTagsToInclude(int templateId) { if (templateId == 111) { // MDSecurityList boolean[] tagsWeAreInterested = getHeaderTags(); includeTags(tagsWeAreInterested, 393, NoRelatedSym, 893, Symbol, SecurityID); return tagsWeAreInterested; } if (templateId == 126 || templateId == 123) { // MDIncRefresh boolean[] tagsWeAreInterested = getHeaderTags(); includeTags(tagsWeAreInterested, NoMDEntries, MDEntryType, MDUpdateAction, SecurityID, OrderID, MDEntryPositionNo, MDEntryPx, MDEntrySize, TradeCondition, MDEntryBuyer, MDEntrySeller); return tagsWeAreInterested; } if (templateId == 120 || templateId == 100 || templateId == 49) { // News boolean[] tagsWeAreInterested = getHeaderTags(); includeTags(tagsWeAreInterested, 148, NoRelatedSym, SecurityID); return tagsWeAreInterested; } return getAllTags(); } };
Note that if you don’t want to specify the tags for a template you must return getAllTags()
so all tags are included in the parsed fix message.
Conclusion
CoralFIX supports the FAST protocol through an auto-generated, ultra-fast, garbage-free, JIT-friendly FAST decoder. You don’t have to understand or care about the FAST protocol details. All you have to do is let CoralFIX generate from the exchange XML templates some JIT-friendly source for you and use it to translate FAST bits into a fully functional FixMessage
object.
