Some exchanges (like CME and BMF/Bovespa) use the FAST protocol for market data streaming. The FAST protocol turns FIX messages into compressed binary and vice-versa saving a lot of bandwidth. In this article we show how CoralFIX takes care of all FAST complexity and translates FAST bits into a FIX message at great speed, without producing any garbage and with auto-generated JIT-friendly source code to ensure maximum performance and low variance. In a following article we will present our FAST decoder benchmarks.
The FAST Template
The exchanges provide their FAST templates in XML format so that you can know how to decode their market data FAST stream. For an example of a XML template, you can click here for the BMF/Bovespa templates. Below we list the template for the FIX message MDIncRefresh_126:
<template xmlns="http://www.fixprotocol.org/ns/fast/td/1.1" name="MDIncRefresh_126" id="126" dictionary="126">
<!-- desc="PREVIOUS VERSION WAS 123" -->
<string name="ApplVerID" id="1128">
<constant value="9" />
</string>
<string name="MessageType" id="35">
<constant value="X" />
</string>
<uInt32 name="MsgSeqNum" id="34" />
<uInt64 name="SendingTime" id="52" />
<uInt32 name="TradeDate" id="75" />
<sequence name="MDEntries">
<length name="NoMDEntries" id="268" />
<uInt32 name="MDUpdateAction" id="279">
<copy value="1" />
</uInt32>
<uInt32 name="MDPriceLevel" id="1023" presence="optional">
<increment />
</uInt32>
<string name="MDEntryType" id="269">
<copy value="0" />
</string>
<string name="Symbol" id="55" presence="optional" />
<uInt32 name="SecurityIDSource" id="22">
<constant value="8" />
</uInt32>
<string name="SecurityExchange" id="207">
<constant value="BVMF" />
</string>
<uInt64 name="SecurityID" id="48">
<copy />
</uInt64>
<uInt32 name="RptSeq" id="83">
<increment />
</uInt32>
<string name="QuoteCondition" id="276" presence="optional">
<default />
</string>
<decimal name="MDEntryPx" id="270" presence="optional">
<exponent>
<default value="-2" />
</exponent>
<mantissa>
<delta />
</mantissa>
</decimal>
<uInt32 name="NumberOfOrders" id="346" presence="optional">
<default />
</uInt32>
<string name="PriceType" id="423" presence="optional" />
<uInt32 name="MDEntryTime" id="273">
<copy />
</uInt32>
<int64 name="MDEntrySize" id="271" presence="optional">
<delta />
</int64>
<uInt32 name="MDEntryDate" id="272" presence="optional">
<copy />
</uInt32>
<uInt32 name="MDInsertDate" id="37016" presence="optional">
<copy />
</uInt32>
<uInt32 name="MDInsertTime" id="37017" presence="optional">
<copy />
</uInt32>
<string name="MDStreamID" id="1500" presence="optional">
<default />
</string>
<string name="Currency" id="15" presence="optional">
<copy />
</string>
<string name="TradingSessionID" id="336" presence="optional">
<default value="2" />
</string>
<decimal name="NetChgPrevDay" id="451" presence="optional">
<exponent>
<default />
</exponent>
<mantissa>
<delta />
</mantissa>
</decimal>
<uInt32 name="SellerDays" id="287" presence="optional">
<default />
</uInt32>
<uInt64 name="TradeVolume" id="1020" presence="optional">
<delta />
</uInt64>
<string name="TickDirection" id="274" presence="optional">
<default />
</string>
<string name="TradeCondition" id="277" presence="optional" />
<uInt32 name="OpenCloseSettleFlag" id="286" presence="optional" />
<string name="OrderID" id="37" presence="optional">
<default />
</string>
<string name="TradeID" id="1003" presence="optional">
<default />
</string>
<string name="MDEntryBuyer" id="288" presence="optional">
<default />
</string>
<string name="MDEntrySeller" id="289" presence="optional">
<default />
</string>
<uInt32 name="MDEntryPositionNo" id="290" presence="optional">
<default />
</uInt32>
<string name="SettlType" id="63" presence="optional">
<default />
</string>
<uInt32 name="SettlDate" id="64" presence="optional">
<default />
</uInt32>
<uInt32 name="SettlePriceType" id="731" presence="optional" />
<string name="PriceBandType" id="6939" presence="optional">
<default />
</string>
<uInt32 name="PriceLimitType" id="1306" presence="optional">
<default />
</uInt32>
<decimal name="LowLimitPrice" id="1148" presence="optional">
<exponent>
<default />
</exponent>
<mantissa>
<delta />
</mantissa>
</decimal>
<decimal name="HighLimitPrice" id="1149" presence="optional">
<exponent>
<default />
</exponent>
<mantissa>
<delta />
</mantissa>
</decimal>
<decimal name="TradingRefPrice" id="1150" presence="optional">
<exponent>
<default />
</exponent>
<mantissa>
<delta />
</mantissa>
</decimal>
</sequence>
</template>
The Generated Source Code
Fortunately with CoralFIX you don’t have to understand or care about this FAST template or the details of the FAST protocol. All you have to do is provide the URL of the exchange XML to CoralFIX and it will generate a low-latency, garbage-free FAST decoder for you. The template above would generate the following source code:
package com.coralblocks.coralmd.parser.bvmf.derivatives.fast;
import static com.coralblocks.coralfix.fast.TypeDecoder.*;
import java.nio.ByteBuffer;
import com.coralblocks.coralbits.NullableStringBuilder;
import com.coralblocks.coralfix.FixMessage;
import com.coralblocks.coralfix.fast.FastTemplate;
import com.coralblocks.coralfix.fast.PMap;
import com.coralblocks.coralfix.fast.field.*;
import com.coralblocks.coralfix.*;
public final class MDIncRefresh_126 implements FastTemplate {
@Override
public int getTemplateId() {
return 126;
}
private final PMap pMap1 = new PMap();
private final FastField field1 = new Constant(1128, false, "9"); // ApplVerID
private final FastField field2 = new Constant(35, false, "X"); // MessageType
private final FastField field3 = new NoOperationUnsignedNumber(34, false); // MsgSeqNum
private final FastField field4 = new NoOperationUnsignedNumber(52, false); // SendingTime
private final FastField field5 = new NoOperationUnsignedNumber(75, false); // TradeDate
private final FastField field6 = new CopyUnsignedNumber(279, false, "1"); // MDUpdateAction
private final FastField field7 = new IncrementUnsignedNumber(1023, true, null); // MDPriceLevel
private final FastField field8 = new CopyString(269, false, "0"); // MDEntryType
private final FastField field9 = new NoOperationString(55, true); // Symbol
private final FastField field10 = new Constant(22, false, "8"); // SecurityIDSource
private final FastField field11 = new Constant(207, false, "BVMF"); // SecurityExchange
private final FastField field12 = new CopyUnsignedNumber(48, false, null); // SecurityID
private final FastField field13 = new IncrementUnsignedNumber(83, false, null); // RptSeq
private final FastField field14 = new DefaultString(276, true, null); // QuoteCondition
private final FastField field15 = new DefaultDeltaDecimal(270, true, -2, 0); // MDEntryPx
private final FastField field16 = new DefaultUnsignedNumber(346, true, null); // NumberOfOrders
private final FastField field17 = new NoOperationString(423, true); // PriceType
private final FastField field18 = new CopyUnsignedNumber(273, false, null); // MDEntryTime
private final FastField field19 = new DeltaSignedNumber(271, true, null); // MDEntrySize
private final FastField field20 = new CopyUnsignedNumber(272, true, null); // MDEntryDate
private final FastField field21 = new CopyUnsignedNumber(37016, true, null); // MDInsertDate
private final FastField field22 = new CopyUnsignedNumber(37017, true, null); // MDInsertTime
private final FastField field23 = new DefaultString(1500, true, null); // MDStreamID
private final FastField field24 = new CopyString(15, true, null); // Currency
private final FastField field25 = new DefaultString(336, true, "2"); // TradingSessionID
private final FastField field26 = new DefaultDeltaDecimal(451, true, NULL_VALUE, 0); // NetChgPrevDay
private final FastField field27 = new DefaultUnsignedNumber(287, true, null); // SellerDays
private final FastField field28 = new DeltaUnsignedNumber(1020, true, null); // TradeVolume
private final FastField field29 = new DefaultString(274, true, null); // TickDirection
private final FastField field30 = new NoOperationString(277, true); // TradeCondition
private final FastField field31 = new NoOperationUnsignedNumber(286, true); // OpenCloseSettleFlag
private final FastField field32 = new DefaultString(37, true, null); // OrderID
private final FastField field33 = new DefaultString(1003, true, null); // TradeID
private final FastField field34 = new DefaultString(288, true, null); // MDEntryBuyer
private final FastField field35 = new DefaultString(289, true, null); // MDEntrySeller
private final FastField field36 = new DefaultUnsignedNumber(290, true, null); // MDEntryPositionNo
private final FastField field37 = new DefaultString(63, true, null); // SettlType
private final FastField field38 = new DefaultUnsignedNumber(64, true, null); // SettlDate
private final FastField field39 = new NoOperationUnsignedNumber(731, true); // SettlePriceType
private final FastField field40 = new DefaultString(6939, true, null); // PriceBandType
private final FastField field41 = new DefaultUnsignedNumber(1306, true, null); // PriceLimitType
private final FastField field42 = new DefaultDeltaDecimal(1148, true, NULL_VALUE, 0); // LowLimitPrice
private final FastField field43 = new DefaultDeltaDecimal(1149, true, NULL_VALUE, 0); // HighLimitPrice
private final FastField field44 = new DefaultDeltaDecimal(1150, true, NULL_VALUE, 0); // TradingRefPrice
@Override
public final void reset() {
field1.reset();
field2.reset();
field3.reset();
field4.reset();
field5.reset();
field6.reset();
field7.reset();
field8.reset();
field9.reset();
field10.reset();
field11.reset();
field12.reset();
field13.reset();
field14.reset();
field15.reset();
field16.reset();
field17.reset();
field18.reset();
field19.reset();
field20.reset();
field21.reset();
field22.reset();
field23.reset();
field24.reset();
field25.reset();
field26.reset();
field27.reset();
field28.reset();
field29.reset();
field30.reset();
field31.reset();
field32.reset();
field33.reset();
field34.reset();
field35.reset();
field36.reset();
field37.reset();
field38.reset();
field39.reset();
field40.reset();
field41.reset();
field42.reset();
field43.reset();
field44.reset();
}
@Override
public final void decode(PMap pMap, ByteBuffer bb, FixMessage fixMsg, boolean[] tagsToInclude) {
boolean[] tags = fixMsg != null ? tagsToInclude : null;
field1.decode(bb, pMap, fixMsg, tags); // ApplVerID (1128)
field2.decode(bb, pMap, fixMsg, tags); // MessageType (35)
field3.decode(bb, pMap, fixMsg, tags); // MsgSeqNum (34)
field4.decode(bb, pMap, fixMsg, tags); // SendingTime (52)
field5.decode(bb, pMap, fixMsg, tags); // TradeDate (75)
int noMDEntries = (int) readUnsignedNumber(bb, false, false);
FixGroup noMDEntriesGroup1 = null;
if (tagsToInclude[268]) noMDEntriesGroup1 = fixMsg.createGroup(268);
for(int i1 = 0; i1 < noMDEntries; i1++) {
FixGroupElement elem1 = noMDEntriesGroup1 != null ? noMDEntriesGroup1.nextElement() : null;
pMap1.read(bb);
boolean[] tags1 = elem1 != null ? tagsToInclude : null;
field6.decode(bb, pMap1, elem1, tags1); // MDUpdateAction (279)
field7.decode(bb, pMap1, elem1, tags1); // MDPriceLevel (1023)
field8.decode(bb, pMap1, elem1, tags1); // MDEntryType (269)
field9.decode(bb, pMap1, elem1, tags1); // Symbol (55)
field10.decode(bb, pMap1, elem1, tags1); // SecurityIDSource (22)
field11.decode(bb, pMap1, elem1, tags1); // SecurityExchange (207)
field12.decode(bb, pMap1, elem1, tags1); // SecurityID (48)
field13.decode(bb, pMap1, elem1, tags1); // RptSeq (83)
field14.decode(bb, pMap1, elem1, tags1); // QuoteCondition (276)
field15.decode(bb, pMap1, elem1, tags1); // MDEntryPx (270)
field16.decode(bb, pMap1, elem1, tags1); // NumberOfOrders (346)
field17.decode(bb, pMap1, elem1, tags1); // PriceType (423)
field18.decode(bb, pMap1, elem1, tags1); // MDEntryTime (273)
field19.decode(bb, pMap1, elem1, tags1); // MDEntrySize (271)
field20.decode(bb, pMap1, elem1, tags1); // MDEntryDate (272)
field21.decode(bb, pMap1, elem1, tags1); // MDInsertDate (37016)
field22.decode(bb, pMap1, elem1, tags1); // MDInsertTime (37017)
field23.decode(bb, pMap1, elem1, tags1); // MDStreamID (1500)
field24.decode(bb, pMap1, elem1, tags1); // Currency (15)
field25.decode(bb, pMap1, elem1, tags1); // TradingSessionID (336)
field26.decode(bb, pMap1, elem1, tags1); // NetChgPrevDay (451)
field27.decode(bb, pMap1, elem1, tags1); // SellerDays (287)
field28.decode(bb, pMap1, elem1, tags1); // TradeVolume (1020)
field29.decode(bb, pMap1, elem1, tags1); // TickDirection (274)
field30.decode(bb, pMap1, elem1, tags1); // TradeCondition (277)
field31.decode(bb, pMap1, elem1, tags1); // OpenCloseSettleFlag (286)
field32.decode(bb, pMap1, elem1, tags1); // OrderID (37)
field33.decode(bb, pMap1, elem1, tags1); // TradeID (1003)
field34.decode(bb, pMap1, elem1, tags1); // MDEntryBuyer (288)
field35.decode(bb, pMap1, elem1, tags1); // MDEntrySeller (289)
field36.decode(bb, pMap1, elem1, tags1); // MDEntryPositionNo (290)
field37.decode(bb, pMap1, elem1, tags1); // SettlType (63)
field38.decode(bb, pMap1, elem1, tags1); // SettlDate (64)
field39.decode(bb, pMap1, elem1, tags1); // SettlePriceType (731)
field40.decode(bb, pMap1, elem1, tags1); // PriceBandType (6939)
field41.decode(bb, pMap1, elem1, tags1); // PriceLimitType (1306)
field42.decode(bb, pMap1, elem1, tags1); // LowLimitPrice (1148)
field43.decode(bb, pMap1, elem1, tags1); // HighLimitPrice (1149)
field44.decode(bb, pMap1, elem1, tags1); // TradingRefPrice (1150)
}
}
}
CoralFIX generates garbage-free and JIT-friendly (Hotspot Just-in-Time compiler) source code to ensure maximum performance and low variance.
Generating the Source Code
To generate source code for all templates from an exchange XML file, you can use the code below. Just run it and a decoder for each message will be generated inside the specified directory and package:
import com.coralblocks.coralfix.fast.TemplatesCodeGenerator;
public class GenerateTemplates {
public static void main(String[] args) throws Throwable {
TemplatesCodeGenerator g = new TemplatesCodeGenerator();
String url = "ftp://ftp.bmf.com.br/FIXFAST/templates/Production/templates-UMDF-NTP.xml";
String outputDir = "./src/main/java/com/coralblocks/coralmd/parser/bvmf/derivatives/fast";
String outputPackage = "com.coralblocks.coralmd.parser.bvmf.derivatives.fast";
String parserName = "BmfFastParser";
g.parseXML(url, outputDir, outputPackage, parserName);
}
}
Notice that it will even create a parser class com.coralblocks.coralmd.parser.bvmf.derivatives.fast.BmfFastParser for you!
Using the Parser
Once the source code is generated you can start using the parser right away to parse FAST bits into FIX messages:
// instantiate the parser FastParser fastParser = new BmfFastParser(); // parse the FAST bits from a ByteBuffer FixMessage fixMsg = fastParser.parse(byteBuffer);
Excluding tags we don’t care
Usually a market data FIX message comes with many FIX tags that we don’t care. To save time during the FAST decoding process, we can specify the tags that we care and the FAST parser will not waste time parsing the other tags. As a result, the parsing will be much faster and the resulting FixMessage object will only have the tags we care about. You specify tags by message type, in other words, by the FAST template id as the example below shows:
private final FastParser fastParser = new BmfFastParser() {
@Override
protected boolean[] getTagsToInclude(int templateId) {
if (templateId == 111) {
// MDSecurityList
boolean[] tagsWeAreInterested = getHeaderTags();
includeTags(tagsWeAreInterested, 393, NoRelatedSym, 893, Symbol, SecurityID);
return tagsWeAreInterested;
}
if (templateId == 126 || templateId == 123) {
// MDIncRefresh
boolean[] tagsWeAreInterested = getHeaderTags();
includeTags(tagsWeAreInterested, NoMDEntries, MDEntryType, MDUpdateAction, SecurityID, OrderID, MDEntryPositionNo, MDEntryPx, MDEntrySize, TradeCondition, MDEntryBuyer, MDEntrySeller);
return tagsWeAreInterested;
}
if (templateId == 120 || templateId == 100 || templateId == 49) {
// News
boolean[] tagsWeAreInterested = getHeaderTags();
includeTags(tagsWeAreInterested, 148, NoRelatedSym, SecurityID);
return tagsWeAreInterested;
}
return getAllTags();
}
};
Note that if you don’t want to specify the tags for a template you must return getAllTags() so all tags are included in the parsed fix message.
Conclusion
CoralFIX supports the FAST protocol through an auto-generated, ultra-fast, garbage-free, JIT-friendly FAST decoder. You don’t have to understand or care about the FAST protocol details. All you have to do is let CoralFIX generate from the exchange XML templates some JIT-friendly source for you and use it to translate FAST bits into a fully functional FixMessage object.
Article
Name
FAST support with CoralFIX
Company
Coral Blocks
Summary
CoralFIX supports the FAST protocol through an auto-generated, ultra-fast, garbage-free, JIT-friendly FAST decoder. You don't have to understand or care about the FAST protocol details. All you have to do is let CoralFIX generate from the exchange XML templates some JIT-friendly source for you and use it to translate FAST bits into a fully functional FixMessage object.