TESSERACT

Processing Software

1

WBS 1.9.8 Structure

2

Processing Flow

3

Standard Tools

4

​

• Data written and stored to HDF5 format
• Common and compressible

​

• Python is standard language across the board
• User friendly, fast, native big data and ML support

​

• Configurations in YAML

​

• Vaex Dataframe
• Designed for TB scale data with high efficiency

VAEX Dataframe

5

Processed Feature/RQ data saved in hdf5 file as VAEX dataframes (https://vaex.io)

Similar syntax as pandas dataframes but with

some advantages:

​

• Out-of-core data handling: Easily work with datasets larger than RAM, zero memory copy policy
• Lazy evaluations: Calculations are deferred until needed, leading to efficient execution
• Selection/cut management
• Fast plotting/visualization
• Unit handling (astropy.units)

Raw Data Reader

• I/O for HDF5 raw data
• Acquire run specific metadata
• Connected channels
• Bias parameters
• Run parameters
• Data type
• Handles HDF5 reading for live pulse display
• Playback of full HDF5 file of data for chosen channels

6

YAML Config Files

• YAML config files are flexible and human readable
• YAML file sets parameters for
• Salting
• Triggering
• Feature extraction
• Trigger channels, windows and threshold
• Feature names, windows, and filters

​

7

Offline Trigger

• Triggering done via NxM optimum filter
• Useable for many channels
• Variables tunable in config file
• Event window size
• Pile up window
• Coincident window
• Template
• Noise spectrum
• Can trigger million events in ~20 minutes
• Dataframe output
• Contains metadata useful for feature extraction
• Developing NxMx2 for HeRALD

8

Feature (RQ) Extraction

• Extraction of relevant pulse and event parameters from the trigger metadata
• Analysis derived pulse templates integrated into processing
• Provides information on event window and pulse parameters
• Baseline
• Chi-square
• Final output in form of analysis ready dataframe
• Easy to add more features

9

Salting

• Inject fake data pulses pre trigger
• Integration from analysis for templates and noise
• Generates user defined number of salt
• Injection is done in-line
• Does not duplicate raw dataset
• Vaex DF output that contains salt parameters
• Raw data with salt is passed directly to trigger
• Investigating use as potential bias mitigation
• Would reduce overall data quantity versus restricted datasets

10

Before

After

Release Management

• Collaboration github for integration and management
• Package specific repos
• Analysis specific repos
• Coordinated tagged releases following large changes/papers

11

Documentation and Training

• Each package contains various tutorials to help new users
• Documentation and READMEs posted on the github with every package
• Writing documentation ongoing process
• Usually handled by developers of specific package

12

Current Status

• Processing software is mostly complete!
• Some updates to integrate Ge/Si detectors needed
• Software is Dark Matter search ready!
• Already used to produce first collaboration results, out very soon!
• Integrations between analysis and processing software well exercised

13

Plot above used every tool described so far. A true processing and analysis effort!

PRELIMINARY

WBS 1.9.8 Milestones

• April/May 2025 CDR Document:
• Review processing performance and scalability
• Review requirements and architecture for Ge/Si devices

​

• Summer 2025 - Early 2026
• Implement modifications (if needed) and test with 4-8 channels
• Test processing at NERSC and CCIN2P3

​

• February 2026
• Processing Readiness Review for Commissioning at Grenoble/Berkeley

​

• Nov 2026
• Processing Readiness Review for LSM (24 channels)

​

14