Cluster Timers

Aeron's cluster timers provide developers with timers that can be safely used within the consensus module. The TimerEvents are appended to the cluster log, ensuring that they will fire correctly during replay events.

Info

Timers within Aeron Cluster are deadline timers and not suitable for interactions requiring exact timing.
Timers are implemented with a TimerWheel data structure. The data structure efficiently manages a large quantity of timer creations and cancelations

The cluster service will ensure that the timer will be called no-sooner-than the scheduled time. For example, if you were to schedule a timer with a deadline of 09:00:00.000, it may be called at 09:00:00.003 - but would never be called at 08:59:59.999 or before. Since timers are played back via the cluster log, and the cluster log is sequentially processed, it is possible that other commands or some existing long running command processing taking place could delay the firing of the timer event.

Note that timers are specific to Aeron cluster – the RAFT protocol has no concept of timers.

Why would you need a timer within a cluster?¶

Aeron Cluster timers are used for both business and technical reasons.

With business events, it is typically used within a state machine transition;
For technical events, such as publishing a cluster event at a specific time or at a given interval.

Within typical OMS or market place services, Aeron cluster timers are being used for:

performing timer based transitions in a workflow, for example, if an RFQ has to be responded to within 5 minutes, you could set a timer to fire at that 5 minute time and have the RFQ automatically transition to an expired state if it hasn't been responded to;
performing specific tasks at a time no-sooner-than-specified, for example, if you need to compute the final value of a trade based upon the prevailing value of a piece of market data at a given time, for example 15:00:00.000, you could schedule a timer to fire at 15:00:00.000 to perform that computation. Note that the market data source would normally be given the expected time $15:00:00.000 in the example$ when retrieving the market data at the point in time, and not the actual cluster time which might be slightly later;
controlling the sending of heartbeats to external systems or user interfaces;
controlling the refresh rate of outbound data. For example, in very active negotiations, it can be more efficient to push all the updates that happened within a given time period $200ms for example$ , rather than streaming out an update for every action.

Aeron cluster timers should not be used for exact timing such as closing a market place. If anything requires exact timing, it is better to rely on inbound cluster messages - either a specific message at the expected time, or have the cluster logic continuously check the cluster time of each inbound message.

Cluster timer API¶

All interactions with the timer API make use of a 64bit long correlation ID. As the developer, you're free to use these 64 bits as you wish - for example, it is not uncommon for two 32 bit integers to be packed into the correlation ID, or for the upper bits of the long to be used to provide a 'namespace' for services that may otherwise generate the same correlation ID.

Usage Considerations

Reusing timer correlation IDs is strongly discouraged. If you reuse a timer correlation ID, any existing active timer using the same correlation ID will be canceled.

Using Long.MAX_VALUE for either a deadline time or correlation ID is not supported as they have specific meanings within Aeron cluster.

Timers should only be scheduled within the core session events onSessionMessage, onSessionOpen, onSessionClose or onTimerEvent. Other usages may be unreliable.

All operations are asynchronous, and races are possible. For example, if attempting to cancel a timer at a point in time very close to the scheduled deadline, it can happen that the onTimerEvent fires before the cancelTimer request was processed.

Remember that it is unsafe to access resources such as an external clock from within a clustered service container

There are three main interactions developers have with the cluster timer:

Managing Timers using Cluster¶

scheduleTimer, which is used to create a new timer. Two arguments are required: the deadline timestamp $in epoch milliseconds$ , and a correlation ID. The results can a true or false, with a true value indicating that the timer was successfully scheduled.
cancelTimer, which can be used to cancel a timer. One argument is required - the correlation ID. The call returns a true if successfully canceled, or a false if it failed.

Timer Events on ClusterService¶

onTimerEvent, which is called on the ClusteredService implementation at a time no-sooner-than the deadline for the given correlation ID. Receives the correlation ID and the timestamp at which the timer expired.

How timers are implemented within Aeron Cluster¶

At the heart of the Aeron cluster Timer Service is an Agrona DeadlineTimerWheel. DeadlineTimerWheel provides efficient storage and execution of timers.

As a part of the ConsensusModule cluster duty cycle, the timer wheel is polled, but only if the node is the leader. When timer events are fired, they are appended to the log as a TimerEvent and are replicated to follower nodes. Once replicated, all nodes receive the TimerEvent and execute the necessary commands.

Info

Different physical nodes will have different clocks that may or may not be in sync. As a result, Aeron cluster can only trust one clock - the current leader's clock.

Timers are managed by Aeron cluster within the ConsensusModule. When a ClusteredService wishes to schedule or cancel a timer, it creates the appropriate SBE message $<code>ScheduleTimer</code> and and <code>CancelTimer</code> respectively respectively$ and offers it on the IPC channel to the ConsensusModule . Being an asynchronous IPC interaction, races are possible between timer execution and management events such as cancellation or rescheduling with a duplicate correlation ID.

Timers and cluster snapshotting¶

Aeron automatically includes any active timers when snapshotting, and any active timers will be included in the snapshot.

Note that you cannot exclude any timers from snapshots, so be sure that the clustered service code can handle all the timers you created. For example, if you elect not to snapshot some cluster state, and that state has timers relating to objects you created but did not rehydrate from a snapshot, your code should not be surprised to receive the timer events.

Timer behavior during log replay¶

Timer events, along with replacement/cancellation and scheduling of new timers are captured in the log. Therefore, all events that are still within the log $i.e. not removed due to snapshotting$ are executed.

Best Practices¶

Caution should be used with using timers to break apart a long running process too aggressively. For example, if you're using timers to process a large set of tasks within the cluster, setting the deadline time to the current cluster time can result in the timer messages taking preference over externally sent commands, reducing the responsiveness of the cluster. If the task cannot be pushed external to the cluster nodes, schedule the tasks with at least a millisecond or more between iterations.

As with all other Aeron publication offer scenarios, the offering of a ScheduleTimer or CancelTimer can fail due to an admin action or back pressure. By default the ClusteredService $via the <code>ConsensusModuleProxy</code>$ will retry three times, however the call to scheduleTimer or cancelTimer may still fail and return a false value. Consider retrying in the case of failures.

In theory, the reuse of correlation IDs plus the cancel/replacement functionality of timers seems useful, but the risk of issues during replay and race conditions close to expiry make it rarely useful and at times harmful. Rather use unique correlation IDs and manage them against internal identifiers within your clustered logic.

When canceling timers - even with unique correlation IDs - you may receive the timer firing event if the cancellation was requested very close to expiry. Ensure that your business logic can deal with this scenario.

Alternative¶

Sometimes messages just need to be run 'as soon as possible, but not before currently sequenced and pending items'. If this matches your requirement, you can achieve this without scheduling a timer.

The offer API on the cluster object provided during onStart offers precisely this capability. When you offer a message via this API, it is sequenced and appended on the log just like any client message would be, although the cluster session is set to null. Under the covers, this is a standard Aeron Publication, so the usual rules around retry during Admin Action, etc., apply.

Note that you cannot call offer (or scheduleTimer) within onStart, onRoleChange or onTerminate. If you try to do this, a ClusterException with the message ERROR - sending messages or scheduling timers is not allowed from x will be raised.

public class YourClusteredService implements ClusteredService
{
    private Cluster cluster;

    ...

    @Override
    public void onStart(Cluster cluster, Image snapshotImage)
    {
        this.cluster = cluster;
        ...
    }  

    public void sendToSelf(DirectBuffer buffer, int offset, int length)
    {
        long result = cluster.offer(buffer, offset, length);
        //retry as needed, especially on result = -3 / ADMIN_ACTION    
    }  
    ...
}