RNTuple Attributes Prototype

Giacomo Parolini

ROOT Team, EP/SFT Group, CERN

RNTuple Attributes - overview

• RNTuple Attributes are a way to associate metadata to an RNTuple
• They are organized in “Attribute Sets”, each stored as one RNTuple on disk
• A regular RNTuple can have 0 or more Attribute Sets associated
• Attribute Sets can’t have Attributes themselves
• The association is done in the “main” RNTuple’s Footer
• “Main” here means the RNTuple that owns the Attribute Sets
• The Footer contains a list of locators to all linked Attribute Sets, keyed by name

2

RNTuple Attributes - overview

3

Main

RNTuple

Footer

AttributeSet�“A”

Anchor

AttributeSet�“B”

Anchor

AttributeSet�“C”

Anchor

Locator for A

Locator for B

Locator for C

AttributeSet Model

• In order to “namespace” the user-provided Attribute field names, we group them under an Untyped Record field

4

Field Zero

<AttrSet_Name>

_rangeStart

_rangeLen

UserAttr1

UserAttr2

Internal Model used by the AttributeSet

User-provided Model

Untyped Record

Writing

5

// Step 5: close attribute range (not optional: the writer will NOT commit the attribute range automatically)

attrSet->CommitRange(std::move(attrRange));

6

////// WRITING RNTuple + Attributes //////

​

// Step 0: create an RNTuple as usual

auto model = RNTupleModel::Create();

auto pInt = model->MakeField<int>("int");

auto file = std::unique_ptr<TFile>(TFile::Open(fileGuard.GetPath().c_str(), "RECREATE")); // NOTE: we need TFile-based, so we

auto writer = RNTupleWriter::Append(std::move(model), "ntpl", *file); // use RNTupleWriter::Append()

// Step 1: create model for the attribute set

auto attrModel = RNTupleModel::Create();

auto pMyAttr = attrModel->MakeField<std::string>("myAttr");

// Step 2: create the attribute set from the writer. Returns a (non-owning) RNTupleAttrSetWriterHandle

// with the same lifetime as the main RNTupleWriter.

auto attrSet = writer->CreateAttributeSet(std::move(attrModel), "MyAttrSet");

// Step 3: open attribute range. attrRange is a RNTupleAttrPendingRange

auto attrRange = attrSet->BeginRange();

// Step 4: assign attribute values.

// Values can be assigned anywhere between BeginRange() and CommitRange().

*pMyAttr = "This is a custom attribute";

for (int i = 0; i < 100; ++i) {

*pInt = i;

writer->Fill();

}

Writing API - some details

• Writing is handled by RNTupleAttrSetWriter. It is an ancillary writer to the main RNTupleWriter (and it’s owned by it) which writes into the same file using a separate FillContext and Sink.
• A RNTupleWriter can have multiple RNTupleAttributeSetWriters at the same time (one per Attribute Set). They all share its lifetime and have a unique name.
• The main public interface consists in BeginRange() and CommitRange()

• Calling BeginRange returns a RNTupleAttrPendingRange, a move-only type that is used as a “transaction token” to pass back to CommitRange.
• In CommitRange the user may pass an entry or implicitly use the default entry of their Model. This API mirrors RNTupleWriter::Fill().

7

RNTupleAttrPendingRange BeginRange();

void CommitRange(RNTupleAttrPendingRange range);

void CommitRange(RNTupleAttrPendingRange range, REntry &entry);

Note: this currently only works for TFile-based RNTupleWriters. Will be extended to support the RMiniFile-based default writer as well.

Lifetime of Attribute Sets and Ranges

8

CreateAttributeSet(“A”)

CloseAttributeSet

RNTupleWriter

BeginRange

CommitRange

CreateAttributeSet(“B”)

AttributeSetWriter�“A”

BeginRange

CommitRange

(destructed)

AttributeSetWriter� “B”

AttrPendingRange

AttrPendingRange

Writing API - cont.

• The API for writing Attributes is similar to that of regular entries:

• Main difference: in the Attributes case we have the concept of a “range” which is represented by the object returned by BeginRange().
• This is only valid between BeginRange() and CommitRange()
• CommitRange() checks that the given range originates from the same AttributeSet
• In both cases the explicit entry used in GetPtr/Fill/CommitRange can be replaced by the default entry of the Model

9

// Regular entries:

auto entry = writer->CreateEntry();

auto ptr = entry->GetPtr<int>("myInt");

*ptr = 42;

writer->Fill(*entry);

// Attributes:

auto attrEntry = attrSet->CreateEntry();

auto range = attrSet->BeginRange();

auto ptr = attrEntry->GetPtr<int>("myInt");

*ptr = 42;

attrSet->CommitRange(std::move(range), *attrEntry);

Writing - some notes

• Attribute ranges may overlap.
• Currently there is no (direct) support for “global attributes”: to set a global attribute one needs to call BeginRange() before any filling and CommitRange() after all entries have been filled.
• An API for setting global attributes on writing is envisioned.
• Empty attribute ranges are supported. When reading, they will only show up when querying all attributes, not for punctual or range-wise queries (see later).

10

Reading

11

auto &attrEntry = attrSet->GetModel().GetDefaultEntry();

for (auto idx : attrs) {

auto range = attrSet->LoadAttrEntry(idx);

auto val = attrEntry.GetPtr<std::string>("string");

EXPECT_EQ(*val, "This is a custom attribute");

EXPECT_EQ(range.Start(), 0);

EXPECT_EQ(range.End(), 100);

}

12

////// READING RNTuple + Attributes //////

​

auto reader = RNTupleReader::Open("ntpl", fileGuard.GetPath());

​

// Fetch a specific attribute set. Returns a unique_ptr<RNTupleAttrSetReader>.

auto attrSet = reader->OpenAttributeSet("MyAttrSet");

​

// NOTE: all these functions return an iterable over indices (ROOT::NTupleSize_t).

​

// Get Attributes of a single entry.

auto attrs = attrSet->GetAttributes(0);

​

// Get Attributes of an entry range

attrs = attrSet->GetAttributesInRange(3, 10);

attrs = attrSet->GetAttributesContainingRange(3, 10);

​

// Get all Attributes

attrs = attrSet->GetAttributes();

Reading - cont.

• Reading is done through the RNTupleAttrSetReader.
• Differently from writing, the AttrSetReader can outlive the main RNTuple reader.
• Reading always happens by iterating over all attribute entries referring to some main entry range; the API resembles the main RNTuple reading API:

13

auto &attrEntry = attrSet->GetModel().GetDefaultEntry();

for (auto idx : attrSet->GetAttributes()) {

auto range = attrSet->LoadAttrEntry(idx);

auto val = attrEntry.GetPtr<int>("int");

}

auto &entry = reader->GetModel().GetDefaultEntry();

for (auto idx : reader->GetEntryRange()) {

reader->LoadEntry(idx);

auto val = entry.GetPtr<int>("int");

}

• Note: LoadAttrEntry can optionally take a REntry as its second argument just like RNTupleReader::LoadEntry.

Reading - empty ranges

• When reading an AttributeSet containing empty attribute ranges, those ranges only show up when doing “catch-all” queries via GetAttributes() with no arguments.
• When calling GetAttributes(idx), GetAttributesInRange(start, end) or GetAttributesContainingRange(start, end), empty ranges never show up, even if they were opened and closed “within” the queried range.
• This is because logically they don’t belong to any range or index and are not associated to any physical entry in the RNTuple. They are special and should be treated specially by the user code.

14

Merging

15

Merging

• RNTuple Attributes are merged alongside their main RNTuple.
• The resulting RNTuple will have the union of all Attribute Sets of all sources.
• Attribute Sets with the same name in different sources must have compatible schemas (currently they must be identical - may be relaxed in the future).
• The RNTupleMerger/TFileMerger can be instructed to ignore (drop) all Attributes.
• No squashing happens during merging, automatic or otherwise. We might provide it in the future in some form.

16

Thank you!

17

Merging - empty ranges

Say we’re merging AttrSets A, B and C, each with 1 attribute spanning all entries

The merged AttrSet looks like this:

18

A

C

B

10 entries

10 entries

0 entries

A

C

20 entries

Attribute Entries � (name: {start, length})

• A: {0, 10}
• B: {0, 0}
• C: {0, 10}

Attribute Entries � (name: {start, length})

• A: {0, 10}
• B: {10, 0}
• C: {10, 10}

Merging only modifies the entries’ start