NOMAD Plug-ins:�Custom Excel Parser

FAIRmat tutorial 9, April 26th 2023

Amir Golparvar

https://de.freepik.com/vektoren-kostenlos/illustration-des-datenextraktionskonzepts_12079896.htm#query=excel&position=38&from_view=search&track=robertav1_2_sidr

Content

• Excel as an ancient, yet super common data repository
• …on Existing NOMAD Tabular-Parser
• Limitations of the Built-in Tabular Parser
• New Plugin: My-Lab-Excel-Parser
• Demo
• Wrap-up

FAIRmat

• FAIRmat tutorial 9, April 26th 2023

Excel as an ancient, yet super common data repository

FAIRmat tutorial 9, April 26th 2023

Amir Golparvar

Excel as an ancient, yet super common data repository

Excel, a widely used spreadsheet software, is shipped with some useful built-in capabilities like:

• Excel provides some tools for organizing, managing, and analyzing data
• Users can create customized spreadsheets to store, sort, and filter data
• Visualization tools
• Shallow structured data-warehouse
• making data FAIR

FAIRmat

• FAIRmat tutorial 9, April 26th 2023

…on Existing NOMAD Tabular-Parser

FAIRmat tutorial 9, April 26th 2023

Amir Golparvar

…on Existing NOMAD Tabular-Parser

NOMAD is shipped with a built-in tabular-parser to parse a given Excel/csv file which offers 4 modes of action:

​

• Column mode:

parsing an entire column into a Quantity

• Row mode:

parsing as many (sub)sections as there are rows in the excel sheet

• Entry mode:

parsing as many entries as there are rows in the excel sheet

• Mixed mode:

parsing using both Column and Row modes

FAIRmat

• FAIRmat tutorial 9, April 26th 2023

Example: Column mode

definitions:

name: 'Tabular Parser example schema'

sections:

Tabular_Parser:

base_sections:

- nomad.parsing.tabular.TableData

quantities:

data_file:

default: datafile.xlsx

type: str

m_annotations:

tabular_parser:

comment: '#'

mode: column

My_Quantity:

type: str

shape: ['*']

m_annotations:

tabular:

name: My_Sheet/My_Column_1

data:

m_def: Tabular_Parser

data_file: datafile.xlsx

1

2

3

Excel File

Schema File

Parsed data

FAIRmat

• FAIRmat tutorial 9, April 26th 2023

…on Existing NOMAD Tabular-Parser

Find out more on tabular parser from documentations here:

• https://nomad-lab.eu/prod/v1/staging/docs/schema/elns.html

​

Follow a tutorial on how to develop schemas using tabular parser here:

• https://github.com/FAIRmat-NFDI/AreaA-Examples/tree/main/tutorial8/5_tabular_parser

​

FAIRmat

• FAIRmat tutorial 9, April 26th 2023

Limitations of the Built-in Tabular Parser

FAIRmat tutorial 9, April 26th 2023

Amir Golparvar

Limitations of the Built-in Tabular Parser

• Always require a (column) header to match and map the data to a quantity
• Cannot parse a single cell/range of cells

quantities:

Atomic_Number:

type: str

shape: ['*']

m_annotations:

tabular:

name: Z

Element_Symbol:

type: str

shape: ['*']

m_annotations:

tabular:

name: Sym

Element_Name:

type: str

shape: ['*']

m_annotations:

tabular:

name: Element

1

Excel File

2

Schema File

FAIRmat

• FAIRmat tutorial 9, April 26th 2023

Limitations of the Built-in Tabular Parser

• Always require a (column) header to match and map the data to a quantity
• Cannot parse a single cell/range of cells

FAIRmat

• FAIRmat tutorial 9, April 26th 2023

New Plugin: My-Lab-Excel-Parser

• Parsing single cell/range of cells from an excel file

Excel-Parser-Example

│

│──excelparser

│ ├── __init__.py

│ ├── nomad_plugin.yaml

│ └── parser.py

│──tests

│ ├── data

│ | └── test.my-lab-excel-file.xlsx

│ └── test-parser.py

│── LICENSE

│── README.md

│── nomad.yaml

└── requirements.txt*

plugin_type: parser

name: parsers/excel_parser

description: |

This is a simple excel parser.

mainfile_name_re: ^.*\.my-lab-excel-file\.xlsx$

mainfile_mime_re: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'

parser_class_name: excelparser.MyLabExcelParser

code_name: Example code

(1)

Plugin Folder Structure

(2)

nomad_plugin.yaml

!

FAIRmat

• FAIRmat tutorial 9, April 26th 2023

Demonstration

FAIRmat tutorial 9, April 26th 2023

Amir Golparvar

Thank you!

• With great flexibility comes great responsibility!
• Documentation saves lives!
• You can ask later question through our forums, GitHub issues, or support email: https://nomad-lab.eu/nomad-lab/help.html�
• The source code can be found under:�https://github.com/amirgolp/tutorial-9-excel-parser�
• Are there questions?

FAIRmat

• FAIRmat tutorial 9, April 26th 2023