Skip to content

Overview

Aeron Archive enables the recording and replay of streams from durable storage.

Architecture

archive Receiving Host Sending Host Primary Data Flow Control Response Channel Control Request Channel Replay Channel File I/O Aeron Archive Aeron Subscription Aeron Publication File System Aeron Archive Client

Aeron Archive adds in two additional components:

  • Aeron Archive, which resides on the process persisting and replaying streams, and
  • Aeron Archive Client, which resides on the process receiving archived streams.

Once the Aeron Archive and Aeron Archive Client are connected, the client process requests a replay of a given stream to a target subscription. This subscription does not need to be in the same process, or even on the same host.

Key features

  • Aeron can simulate subscriptions, allowing publications to be recorded without an active subscription. This is often very useful for scenarios when the recording process may be running when the subscriber has not yet started, or has failed.
  • Existing recordings can be extended. This is useful following a restart - by allowing an existing Aeron Archive recording to be appended to.
  • Recordings can be replayed from a given recording and a given position in a recording. Replay can be direct from an archive, or can be started on an archive, merged with a real-time stream, and then the subscription continues exclusively from the stream.
  • Clients can query the Aeron Archive process for available recordings - this is useful to detect what is available on the Aeron Archive before starting to replay.
  • Recordings can be truncated and purged. Truncated is used to remove the data after a given position, and purging is used for removing archive segments before a given position. In addition, segments can be attached, detached, and deleted allowing for finer grain management of Aeron Archive data segments.
  • Aeron Archive supports migrating the recording data from version to version (for example from an older version to a newer version of Aeron Archive), enabling the data to be used across Aeron Archive upgrades.
  • Archive recordings can be replicated from a source process to a destination process. This is useful for near real-time backups of data.

Usage overview

Using Aeron Archive is simpler once you understand the configuration and how Aeron Archive communicates.

Let’s consider the above diagram, with two physical hosts: Sending Host and Receiving Host. The Sending Host is using Aeron Archive to record a local publication. The Receiving Host requires a copy of this stream at a later stage, and uses Aeron Archive Client to interact with the remote Aeron Archive.

On start up of the Sending Host, the Aeron Archive opens a subscription on the control channel and stream ID configured in the Aeron Archive context, and begins to poll for inbound requests. While this is happening, the Sending Host opens a recorded publication and begins offering data. Aeron Archive persists this data to disk on the Sending Host.

At some later stage, the Receiving Host is ready to receive the data held in the Sending Host Aeron Archive. The Receiving Host constructs an Aeron Archive Client, with four key parameters set: the control request channel and stream ID and the control response channel and stream ID.

The control request channel and stream ID set in the Aeron Archive Client context on the Receiving Host must point to the configured control channel and stream ID set on the Aeron Archive context on the Sending Host. The control response channel and stream ID is used to open a subscription on the Receiving Host and must be accessible from the publication on the Sending Host so that messages can be delivered to the Receiving Host Aeron Archive Client. This control response channel and stream ID is used for administrative response messages flowing from the Aeron Archive on the to the Sending Host to the Aeron Archive Client on the Receiving Host.

Once connected and ready to replay, the Receiving Host will call the replay method on the Aeron Archive Client. This includes the replay channel and stream ID. The Receiving Host sends this replay channel and stream ID to the remote Sending Host over the control request channel and stream ID, which then results in the remote Aeron Archive connecting to the replay channel and stream ID and sending the data as read from the physical disk. The Receiving Host receives data by polling the subscription on the replay channel and stream ID.

Key Channel Configurations

On Aeron Archive

Item Used to
Control Channel/Stream ID Used by an Aeron subscription. Aeron Archive Client on the Receiving Host opens a publication to this port in order to interact with the Sending Host. Multiple Receiving Host processes may interact with this single Control Channel/Stream ID
Recording Events Channel/Stream ID Used by an Aeron subscription. This is the channel and stream to which recording events are published. Needs to be set either to a MDC or multicast channel URI if dynamic subscribers are required.

If you're replicating from a remote Archive, you will also need to configure the Archive.Context.archiveClientContext(), which is the internal Aeron Archive Client used for replication purposes in addition to the replicationChannel. The Archive.Context.archiveClientContext() must have the correct Aeron Directory name, and the Control Response channel must be specified.

On Aeron Archive Client

Item Used to
Control Request Channel/Stream ID Used by an Aeron publication. Aeron Archive Client opens a publication on this port, in order to send the data to the remote Aeron Archive control subscription. Typically points to the Archive Host.
Recording Events Channel/Stream ID Used by an Aeron subscription. Aeron Archive Client receives data on recording events via this subscription. Typically points to the Archive Host.
Control Response Channel/Stream ID Used by an Aeron subscription. This is the address sent by the client to the Sending Host as a response address. The Aeron Archive Control Session opens a publication to this. Typically point to this media driver.
Replay Channel/Stream ID Used by an Aeron subscription. This is the address sent by the client to the Sending Host as a replay address. The Aeron Archive Replay Session opens a publication to this. Note, because this is opened on the Aeron Archive Host side by the Replay Session, the replay channel and stream ID cannot be aeron:ipc unless the Sending Host and Receiving Host share a Media Driver