Skip to content

Direct Buffer

Note

Agrona makes use of sun.misc.Unsafe and sun.nio.ch.SelectorImpl.selectedKeys which will result in the JVM raising some warning log entries regarding illegal reflective access on startup. To remove the warning add the following JVM args: --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens jdk.unsupported/sun.misc=ALL-UNNAMED.

Agrona by default will make use of the system endianness. If you have components running with different endianness then you should specify the endianness to use when reading or writing when using DirectBuffer.

When interacting with Aeron, message data is passed between the client application and Aeron using the DirectBuffer interface. Agrona's DirectBuffers are similar to the Java NIO ByteBuffer, but with a bit more convenience.

Buffers

There are three implementations of DirectBuffer:

Name Implementation Details
UnsafeBuffer Off-heap fixed size buffer. Requires you to first allocate a ByteBuffer of a given size. Provides best possible performance, but will not be resized if the required capacity exceeds the available capacity (an IndexOutOfBoundsException will be thrown).
ExpandableDirectByteBuffer A direct buffer that is expandable and backed by a direct ByteBuffer. Default size is 128 bytes, but this can easily be set using the constructor ExpandableDirectByteBuffer(int initialSize). If the array needs to expand beyond its current size, a new ByteBuffer is allocated and the contents copied over.
ExpandableArrayBuffer A direct buffer backed with a byte array (i.e., allocated with new byte[size]). When it needs to be resized, a new byte[] is created and the contents are copied over.

Key Concepts

Byte Alignment

In more recent versions of Agrona, byte alignment checks are enforced by calls to verifyAlignment. In some cases, such as Agrona RingBuffers, this is called within the constructor. If there is misalignment, you will receive an error stating AtomicBuffer was created from a byte[] and is not correctly aligned by x. Misalignment can lead to issues like slower memory access, as the system might have to perform extra operations to assemble the data from parts spread over multiple aligned blocks. It can even cause crashes in certain systems (such as Arm based JVMs) that strictly require alignment for some operations such as those supported by the Atomic objects.

In general, ByteBuffer.allocate will not consistently align the data correctly. ByteBuffer.allocateDirect on the other hand, which allocates off-heap memory, will generally run without issue. If you require specific alignment (for example, at 32 bytes for maximum memcpy performance on x86_64), Agrona offers BufferUtil.allocateDirectAligned. This accepts two arguments: the desired length and the alignment boundary required. Alternatively, agrona.strict.alignment.checks can be used to prevent alignment checking.

Byte Order

Agrona allows you to set specific byte ordering, and if not, it will default to the order provided by ByteOrder.nativeOrder(). Reading and writing data with different byte orders will corrupt data. This can happen when crossing platform and/or operating system boundaries.

Offset & Length

If we wanted to extract a 4-byte integer from the below 13-byte buffer that is held within the highlighted section, we would read from offset 4 with a length 4. The offset refers to where in the array to begin reading, and the length refers to how many bytes to read.

Byte Array with 32bits highlighted

This same method is provided by Aeron when reading off of a subscription. The FragmentHandler assigned to the poll of the Subscription will need to implement the following method:

1
void onFragment(DirectBuffer buffer, int offset, int length, Header header);

As you can see, you're provided with the DirectBuffer to read from, plus the offset to read from and the length of bytes to read. Forgetting to use the offset when reading is a common mistake made by those new to Aeron.

Working with Types

Chars & Bytes

DirectBuffer implementations offer methods to read and write both individual bytes along with regions of bytes. Char data is also supported, although it should be noted that the chars are 16 bit chars. For most use cases Byte values are sufficient.

Shorts, Integers & Longs

DirectBuffer implementations offer methods to read and write short, int and long values.

For int and long, Agrona offers additional methods that allow the reading and writing of natural (ASCII formatted) values. UnsafeBuffer also adds compare-and-set, get-and-add and get-and-set functionality.

Unsafe extras

Get and Add

This method returns the existing value and adds the provided delta to the value in the unsafe buffer.

1
2
3
4
5
6
//place 41 at index 0
unsafeBuffer.putLong(0, 41);
//add 1 to the long at index 0 and return the old value, which is 41
long originalValue = unsafeBuffer.getAndAddLong(0, 1);
//read the value of the long at index 0; is now 42
long plus1 = unsafeBuffer.getLong(0);

Get and Set

This method returns the existing value and sets the value in the unsafe buffer to the provided value.

1
2
3
4
5
//read current value while writing a new value.
//Assuming code continues from above, oldValue = 42
long oldValue = unsafeBuffer.getAndSetLong(0, 43);
//read the value of the long at index 0, which is now 43
long newValue = unsafeBuffer.getLong(0);

Compare and Set

This method checks if the value in the buffer was the expected value provided. If it was the expected value, the new value provided is placed into the buffer and the method returns true. If it was not the expected value, the method returns false and the buffer value is unchanged.

1
2
3
4
5
//check the value was what was expected, returning true/false if it was. Then update the value a new value
//Assuming code continues from above, wasExpected = true
boolean wasExpected = unsafeBuffer.compareAndSetLong(0, 43, 44);
//read the value of the long at index 0, which is now 44
long updatedValue = unsafeBuffer.getLong(0);

See GitHub for a test showing usage of these.

Floats & Doubles

Note

Using floats and doubles are generally not recommended. Either use a BigDecimal formatted to string, and move the string around in the DirectBuffer, or make use of a scaled long (one example of a scaled long is DecimalFloat in Artio)

DirectBuffer implementations offer methods to read and write float and double values.

Strings

There are two general classes of String methods: putStringX and putStringWithoutLengthX (where X is Ascii or Utf8). The WithoutLength variant is used for fixed length strings, while the other accepts variable length strings.

When using variable length string variants, Agrona writes the string as follows:

  • First a integer value is written with the length in bytes;
  • and then the actual string content is written.

An example of a putStringAscii with a 12 byte ASCII string, with byte 0 holding the length, and bytes 1 to 12 holding the content:

String With Length

An example of a putStringWithoutLengthAscii with the same 12 byte ASCII string, with bytes 0 through 11 holding the content:

String no Length

When reading data from the DirectBuffer:

  • it's vital to use the same method for reading as was used to write the string;
  • variable length strings can be less efficient as reading the entire buffer is not possible without knowing the location and length of all the variable length strings. To improve efficiency, either use fixed length strings or pack the variable length strings together at the end of the DirectBuffer.