Lessons Learned Generation of Scientific Text From Data

Based on discussions with the AuroraGPT Data Team, and other AuroraGPT Team members

Robert Underwood

AuroraGPT Data Team co-Lead

Assistant Computer Scientist

Argonne National Laboratory

​

Image Credit: Google Gemini

1

The AuroraGPT Data Team Contributors 2024

2

co-Leads

Scientific Text

Deduplication and Middleware

Biology

Climate

Materials

Motivation

Is a model trained on science text and data more effective on science tasks?

• More effective tokenization of domain science concepts?
• Better “domain awareness” for specialized domains?
• Better support for scientific data modalities?

To answer this question we need to prepare science text and data

3

Biodata Narratives

Arham Khan, Clark Cucinell, et al

4

Bacterial and Viral Bioinformatics Resource Center Database

5

Inspiration: Textbooks are all you need

• Paper (Series) from Microsoft that provides the most details on synthetic generation for LLMs of any of the major commercial players
• Argues that data volume and model size are less important than model quality, and presents an approach to synthetically generate high quality data

6

Figure Reference: https://techcommunity.microsoft.com/blog/azure-ai-services-blog/discover-the-new-multi-lingual-high-quality-phi-3-5-slms/4225280

Madlibs

Simple templating or dumps of data from the database can have very high levels of duplication leading to memorization

7

LLM Augmented Templates

Take the MADLIB and ask an LLM to expand it result in better quality, but still room for improvement

8

Best from last slide

86%

Clearer guidance and perspectives

Below, we've provided a record of data obtained from the Bacterial and Viral Bioinformatics Resource Center (BV-BRC), a comprehensive resource for bacterial and viral infectious disease research. BV-BRC provides integrated data, advanced bioinformatics tools, and workflows to support the scientific community in understanding and combating infectious diseases. Our goal is to summarize the record below, providing readers with a thorough explanation of its content in a neutral, scientific tone suitable for a textbook or scientific report. The target audience is individuals with an advanced (post-graduate level) understanding of Biology, Chemistry, and infectious disease research. Prioritize ensuring the factual integrity of this summary by drawing heavily upon the record for information. You need not use every field, particularly those that are not scientifically relevant (for instance, the md5 hash). However, you should prioritize disseminating as much scientifically relevant information as possible from the given data record.

​

​

9

Best from last slide

43%

So are the perspectives different?

Not yet…

• By visual inspection, the differences are washed out by the “boilerplate” that appears in common to both prompts
• We need to deal with “boilerplate” to get this percentage lower

10

Not comparable with previous slide

Does perspective have any effect?

Yes

• While it does not affect duplicate % because of the boilerplate
• But it is doing something meaningful semantically
• We’ll build on this later

11

Educator Field Usage %

Virologist Field Usage %

The “locus” field appears much more often in virologist generations

Does the LLM Ignore Things?

Yes

• Some of them we don’t care about (e.g. MD5 hash)
• Some of them we do care about (e.g. feature word)

​

We should adjust prompts to ensure critical fields are mentioned

12

So how far can we go?

• Some fields are much more duplicated than others
• Some are 100% unique but not useful (e.g. MD5 Hash)

​

13

More Distinct Fields Exact Duplication %

44%

8.7-0.1%

Contextualization of fields with metadata

exp_title: Title of the experiment/study.

patric_id: PATRIC identifier for the gene or protein feature.

bioset_id: Identifier for the specific bioset (comparison within an experiment).

entity_name: Name or description of the gene or protein (e.g., "hypothetical protein").

treatment_type: The type of treatment applied in the experiment (e.g., "iron").

exp_type: Type of experiment performed, such as transcript quantification.

treatment_name: Name of the treatment condition.

z_score: Z-score for the expression level change in the analysis.

strain: Strain of the organism used in the experiment.

​

14

Best from last slide

26%

Unresolved Questions

1. Occasionally, the longer prompts result in the model getting “stuck” with a repeated generation
2. Can we go further still?
1. Probably, the LLM is bringing in context from its weights and not just the provided example from the database
2. We’d ideally like to link this to a RAG system with content from papers for even more variation
3. Are narratives useful outside of fine-tuning?
• We think so, we expect more contextualized data from a database or an agent would improve that too. Richer data should just be better
• We need to measure this

15

Conclusions and Questions

16

Questions?

Conclusions:

• Contextualization and filtering of fields of data for LLMs matters a lot for synthetic data quality
• Validation of scientific data is just as important as validation of end-to-end model performance
• It is critical to scale data processing pipelines to cope with the huge volumes of scientific data even if they are small by comparison
• Fault-Tolerance and efficient recomputation is critical for rapid prototyping and development of large scale pipelines

Contact:

Robert Underwood

runderwood@anl.gov

AuroraGPT Data coLead

Assistant Computer Scientist

​

17

Acknowledgements

AuroraGPT-LDRD: This material is based upon work supported by Laboratory Directed Research and Development (LDRD) funding from Argonne National Laboratory, provided by the Director, Office of Science, of the U.S. Department of Energy under Contract No. DE-AC02-06CH11357.

AuroraGPT-INCITE: An award for computer time was provided by the U.S. Department of Energy’s (DOE) Innovative and Novel Computational Impact on Theory and Experiment (INCITE) Program. This research used resources from the Argonne Leadership Computing Facility, a U.S. DOE Office of Science user facility at Argonne National Laboratory, which is supported by the Office of Science of the U.S. DOE under Contract No. DE-AC02-06CH11357.

DataStates: This material is based upon work supported by the U.S. Department of Energy (DOE), Office of Science, Office of Advanced Scientific Computing Research, under Contract DE-AC02-06CH11357. This research used resources of the Argonne Leadership Computing Facility at Argonne National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy, Office of Science, under contract number DE-AC02-06CH11357

​

​

18