Skip to content

Protocol

The exact choice of messaging and internal data structures should be carefully considered as the time taken to read and write data into and out of the Aeron buffers directly impacts Aeron Cluster throughput and latency.

Static Data Protocol

Instruments

Instruments will hold a minimal set of data - an instrument ID, the minimum quantity and a flag indicating if active or not.

1
2
3
4
5
6
7
Instrument
{
    int id;
    boolean active;
    String cusip;
    long minQuantity;
}

Commands relating to Instrument:

  • AddInstrumentCommand - used to add a new Instrument to the state machine. Note that it includes very limited data - not even a name or otherwise. This is because this is all the state machine needs. Gateways around the edges can translate the instrument ID into something appropriate for external consumption.
  • EnableInstrumentCommand - used to enable or disable the instrument. All RFQs in progress with this instrument would be canceled, and new RFQs for this instrument would fail validation. Two parameters are required: cusip & enabled.

RFQ Interaction Protocol

Since there are good number of possible interactions, only a few samples are included to illustrate the design.

CreateRfqCommand

This is sent by a User to initiate a new RFQ.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
CreateRfqCommand
{
    int correlation;
    long expireTimeMs;
    long quantity;
    String side;
    String clOrdId;
    String cusip;
    int userId;
}

You'll notice that the command include data on the user performing the request. The cluster has no concept of users as they are expected to be fully managed in the gateways.

RfqCreatedEvent

This is broadcast by the state machine following the creation of a new RFQ.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
RfqCreatedEvent
{
    int correlation;
    int securityId;
    long expireTimeMs;
    long quantity;
    String side;
    int rfqId;
    int rfqRequesterUserId;
    String clOrdId;
}

You'll notice this data is almost identical to the CreateRfqCommand. The gateways receiving the broadcasted RfqCreatedEvent can process this data such that the Rfq Requester receives a message specific to their inbound request (with correlation, clOrdId), and everyone else receives a generic 'New RFQ' message (without correlation, clOrdId).

Protocol Performance

JMH results for the roundtrip write/read time for a typical protocol message are below, showing a 99th percentile time of 56ns for the roundtrip. Since the time taken to read and write the commands and events directly impacts the performance of the cluster, it is beneficial to minimise the time spent processing the input commands and output events of the cluster.

Benchmark                   Mode      Cnt       Score    Error   Units
                            sample  3163310      55.119 ±  0.820   ns/op
counterRoundtrip·p0.00    sample                1.000            ns/op
counterRoundtrip·p0.50    sample               47.000            ns/op
counterRoundtrip·p0.90    sample               52.000            ns/op
counterRoundtrip·p0.95    sample               54.000            ns/op
counterRoundtrip·p0.99    sample               56.000            ns/op
counterRoundtrip·p0.999   sample              161.689            ns/op
counterRoundtrip·p0.9999  sample            19562.810            ns/op
counterRoundtrip·p1.00    sample           280064.000            ns/op
·gc.alloc.rate            sample       15       0.025 ±  0.001  MB/sec
·gc.alloc.rate.norm       sample       15       0.001 ±  0.001    B/op
·gc.count                 sample       15          0           counts

Improving Protocol Performance

The RFQ protocol uses strings in several areas - including Side, ClOrdId and CUSIP. Swapping out these strings for short, integer or longs will increase performance of both reading and writing the protocol to and from buffers. This can be done by using identifiers whereever possible, and by shifting logic into the gateways in order to reduce the data going into the cluster to the bare minimum. An additional benefit to remove all strings from the protocol is this reduces Java Garbage Collection. The sample code's protocols are almost entirely String free.

Back to top