1 of 15

Do Protein Transformers Have Biological Intelligence?

¹University of Delaware,

²Beijing University of Posts and Telecommunications,

³Yale University,

⁴University of Louisiana at Lafayette

Fudong Lin¹, Wanrou Du², Jinchan Liu³, Tarikul Milon⁴,

Shelby Meche⁴, Wu Xu⁴, Xiaoqi Qin², Xu Yuan¹

2 of 15

2

Protein Transformers

AlphaFold 2

AlphaFold 3

MSA Transformer

ESM-1v

3 of 15

3

Limitations of Existing Studies

Our community lacks high-quality, expert-annotated scientific datasets.
Existing Protein Transformers lack domain-specific model designs, limiting their effectiveness in protein function predictions.
Existing models operate in a “black-box” manner, making their decision-making processes difficult to interpret.

1. Despite recent progress, existing studies for protein function predictions still face three major limitations.

2. First, although abundant protein data are available in the Protein Data Bank (PDB), the community still lacks high-quality, expert-annotated scientific datasets. Addressing this gap requires close collaboration between AI and biology experts.

3. Second, current Protein Transformers largely adopt architectures from natural language processing (NLP) without incorporating domain-specific design principles. This limits their effectiveness in protein function prediction, forcing models to rely on pre-training with millions of protein sequences to achieve reasonable performance.

4. Third, while interpretable Protein Transformers could yield valuable insights for scientific discovery—such as accelerating drug development—existing models function in a “black-box” manner, offering little transparency into their decision-making processes.

4 of 15

4

Our Motivation

Can Protein Transformers capture biological intelligence embedded in

protein sequences?

Scientific Dataset

Protein Transformer

Explainable AI (XAI)

Curate a scientific dataset with meaningful annotations, tailored for protein function predictions
Devise a new computation-efficient Protein Transformer, lifting the need of large-scale pre-training
Develop a novel XAI technique for decoding decision-making processes of Protein Transformers

1. In this work, we address the following research question: Can Protein Transformers capture biological intelligence embedded in protein sequences?

2. Here, biological intelligence refers to the meaningful patterns identified by Protein Transformers, which align closely with established domain knowledge.

3. To answer this question, we make three key contributions:

4. First, we construct a new dataset of over 9,000 proteins with expert-informed annotations, specifically tailored for protein function prediction.

5. Second, we design a computation-efficient architecture that achieves strong performance when trained from scratch, eliminating the need for large-scale pre-training.

6. Third, we develop a novel Explainable Artificial Intelligence (XAI) technique, which can efficiently interpret the decision-making processes of Protein Transformers. This approach enables more transparent and trustworthy scientific discoveries.

5 of 15

5

Our Protein-FN Dataset

Offer 9K proteins, including their 1D amino acid sequences, 3D protein structures, and functional properties
Useful for various biological tasks, e.g., protein function predictions, motif identification and discoveries, etc.

Dataset Overview

1D Sequence

3D Structure

6 of 15

6

Computation-Effecient Protein Transformers

Amino Acid Embedding: Directly encode biologically meaningful features, lifting the requirement of extensive pre-training
Flexible Positional Embedding: Capture proteins with post-translational modifications or disordered regions

Sequence Protein Transformers (SPT)

Previous Protein Transformers require the pre-training on millions of proteins to achieve decent performance.

However, this pre-training is computationally prohibitive for most academic groups, due to its tremendous computational overhead.

In this work, we develop the Sequence Protein Transformers (SPT) for computation-efficient protein function predictions.

The figure shows its model architecture.

It has two key designs.

First, instead of using the tokenizer in NLP for amino acid encoding, we propose a novel embedding method that directly encode biologically meaningful features.

This design lifts the requirement of computationally extensive pre-training.

Second, we devise a new flexible positional embedding for capturing proteins with post-translational modifications or disordered regions.

As such, our model is efficient and effective in protein function predictions.

7 of 15

7

Our Sequence Score

Explainable AI (XAI): Decode the decision-making processes of deep neural networks (DNNs)
Sequence Score: Given a decision of interest, our approach assigns each amino acid an importance score reflecting its actual contribution to that decision.

Importance Weight:

Importance Score:

Normalization:

Equations

Notations

8 of 15

8

Evaluation on Our SPT Models

Dataset: Protein-FN, 7.2K training samples and 1.8K test samples
Baselines: TAPE, ESM-1b, and ESM-1v
Metrics: GFLOPs for computational overhead and Test Error Rate for prediction performance

Comparison to Protein Transformers on our Protein-FN dataset

Our models are efficient and effective on protein function predictions.

9 of 15

9

Evaluation on Our SPT Models

Dataset: Antibiotic Resistance (AR) and Metal Ion Binding (MIB)
Baselines: TAPE, ESM-1b, and ESM-1v
Metrics: GFLOPs for computational overhead and Test Error Rate for prediction performance

Comparison to Protein Transformers on AR and MIB datasets

Our models can be generalized to common benchmark datasets.

10 of 15

10

Evaluation on Our Sequence Score

The Faithfulness of Our Sequence Score

Metric: Measure the alignment between importance scores assigned to amino acids and their actual impact on the prediction
Deletion Method: Observe performance decrease caused by masking a certain ratio (or number of) amino acids
Mutation Method: Observe performance degradation caused by mutating a certain ratio (or number of) amino acids

Deletion Method

Mutation Method

Our approach assigns high importance scores to amino acids that are essential for accurate predictions.

We evaluate our Sequence Score technique for decoding the decision-making processes of Protein Transformers.

Here, we consider the Faithfulness of our approach.

The Faithfulness metric measures the alignment between importance scores assigned to amino acids and their actual impact on the prediction.

We use the Deletion and Mutation methods to evaluate the Faithfulness.

The Deletion method is to observe performance decrease caused by masking a certain ratio (or number of) amino acids.

The Mutation method is to observe performance degradation caused by mutating a certain ratio (or number of) amino acids.

From the figure, the deletion (or mutation) of amino acids with highest importance scores consistently cause lager performance decrease than those with lowest scores.

Therefore, our approach assigns high importance scores to amino acids that are essential for accurate predictions.

11 of 15

11

Evaluation on Our Sequence Score

Importance Scores for the Receptor (L) and Isomerase (R) Classes

Our approach consistently attributes similar scores to structurally comparable proteins.

The Stability of Our Sequence Score Technique

Metric: Assess the consistency of the importance scores assigned to amino acids with similar structures
Dataset: Receptor and Isomerase proteins obtained from our Protein-FN dataset

12 of 15

12

Interpret Biological Intelligence

Definition and Terminology

Biological Intelligence: Discover meaningful biological patterns, which align with established domain knowledge
Protein Motif: A pattern of amino acids that share among different proteins

Zinc-Binding Motif

“H94-H96-H119”

Catalytic Triad

“H57-D102-S195”

Protein Transformers can capture biological intelligence inherence with protein sequences.

13 of 15

13

Open Source Efforts

Release of Dataset and Code

Hugging Face Datasets: https://huggingface.co/datasets/Protein-FN/Protein-FN
GitHub Repository: https: //github.com/fudong03/BioIntelligence

Hugging Face Datasets

GitHub Repository

14 of 15

14

Conclusion

This work has explored the capabilities of Protein Transformers in capturing biological intelligence resided in protein sequences.

We introduced a high-quality, expert-annotated Protein-FN dataset, a computation-efficient Protein Transformer, and an XAI technique for decoding decision-making processes of Protein Transformers.
Our models are efficient and effective on protein function predictions, and our XAI technique can help reveal biological intelligence captured by Protein Transformers.

Dataset

Code

Paper

15 of 15

15

Thank you!

Q & A