Large Language Models & Biological Foundation Models

Syed Asad Rizvi

vanDijk Lab, Yale University

​

Models, Inference & Algorithms

October 16^th, 2024

State of AI research

Yale University

2

10/16/24

Exciting times to be in AI research!

Emu video: Text 🡪 Video

(Girdhar et al., 2023)

GPT-4

Claude

Llama-3

Gemini

Stable Diffusion v2

(Rombach et al., 2023)

AlphaFold 3

(Abramson et al., 2024)

DALL-E 3

(OpenAI, Goh et al., 2023)

SEED-Llama

(Ge et al., 2023)

OneLLM

(Han et al., 2024)

What are Large Language Models?

Yale University

3

10/16/24

• Large Language Models (LLMs) are very large Deep Learning models trained on text from the internet

Image generated using Dall-E

Recent major LLM releases

What can LLMs do?

Yale University

4

10/16/24

OpenAI. (2024). GPT-4o (Oct 11 version)

Image: Cobus Greyling, Medium, 2023

Performant LLMs

Application Ecosystem

Behind it all: The Transformer

Yale University

5

10/16/24

• LLMs are based on the Transformer architecture (Vaswani et al., 2017)
• Components:
• Multi-head attention blocks
• Feedforward layers
• Layer Normalization
• Skip connections
• Positional encoding

Vaswani et al., NeurIPS 2017

Text tokenization

Yale University

6

10/16/24

• Tokenization: break input text sequences into word pieces
• Typical algorithm: Byte-pair encoding

In GPT-style models, tokenization is the process of converting text into smaller pieces called tokens. These tokens can be words, parts of words, or even individual characters, depending on the model's vocabulary.

Tokenization allows the model to process text in a structured way, breaking down complex sentences into manageable pieces.

Word pieces

[818, 402, 11571, 12, 7635, 4981, 11, 11241, 1634, 318, 262, 1429, 286, 23202, 2420, 656, 4833, 5207, 1444, 16326, 13, 2312, 16326, 460, 307, 2456, 11, 3354, 286, 2456, 11, 393, 772, 1981, 3435, 11, 6906, 319, 262, 2746, 338, 25818, 13, 29130, 1634, 3578, 262, 2746, 284, 1429, 2420, 287, 257, 20793, 835, 11, 7163, 866, 3716, 13439, 656, 36426, 5207, 13]

Dictionary entries

Next token prediction

Yale University

7

10/16/24

• Decoder-only architecture
• Next token prediction: Language modeling head predicts the next word of input sequence

Transformer

decoder

Causal attention masking

Yale University

8

10/16/24

LLM development recipe

9

10/16/24

Andrej Karpathy, State of GPT, 2023

Pretraining stage

Yale University

10

10/16/24

GPT-3 training corpus (Brown et al., 2023)

Supervised finetuning stage

Yale University

11

10/16/24

Summarize the following proposed legislation (bill): SECTION 1. SHORT TITLE. This Act may be cited as the ``Office of National Environmental Technologies Act''. SEC. 2. PURPOSE. It is the purpose of this Act-- (1) to improve coordination and integration of research and development performed by government agencies and that performed through government-awarded contracts and grants; and (2) to assist the efforts of private industry, universities, nonprofit research centers, and government laboratories to provide environmentally safe technical solutions to problems threatening the Nation's environmental security and, in the process, to help the Nation's competitiveness. SEC. 3. DEFINITIONS. For the purposes of ...

The proposed Office of National Environmental Technologies Act aims to improve the coordination and integration of research and development for environmentally safe technical solutions through government agencies, grants, and private industry. It establishes the Office of National Environmental Technologies within the Environmental Protection Agency and an Industry and Academia Advisory Council.

Ideal response

Input prompt (example from GPT-4all, Nomic AI):

Reward modeling

Yale University

12

10/16/24

OpenAI. (2024). GPT-4o (Oct 11 version)

Reward modeling

Yale University

13

10/16/24

1.1

0.4

-0.5

Reinforcement Learning from Human Feedback

Yale University

14

10/16/24

• Policy: a language model that takes in a prompt and returns a sequence of text
• Action space: LLM vocabulary
• Observation space: distribution of input sequences
• Reward function: fixed reward model + KL penalty
• Optimize policy to maximize reward of responses using Proximal Policy Optimization (Schulman et al.)

1.1

0.4

-0.5

Why do reinforcement learning?

Yale University

15

10/16/24

• Works better – aligned more to human-preferred outputs
• Data generation: easier to discriminate between responses than creating optimal ones

Ouyang et al., InstructGPT, NeurIPS 2022

Foundational Ideas for Improving LLMs

Scaling laws for LLMs

Yale University

17

10/16/24

• Performance of LLMs improves following a power law with model size, dataset size, and amount of training compute
• Relationship allows us to predict performance of future LLMs & allocate compute budgets effectively

Scaling laws for Neural Language Models (Kaplan et al., 2020)

Chain-of-Thought: Reasoning step-by-step

Yale University

18

10/16/24

• Generating intermediate output steps improves LLM performance on complex reasoning tasks

Chain-of-Thought Prompting, NeurIPS 2022

Tree of Thoughts, NeurIPS 2024

Scaling compute at inference

Yale University

19

10/16/24

• OpenAI o1: Series of models by OpenAI that spend more time thinking before generating a final response

OpenAI. (2024). OpenAI-o1 (Oct 15 version)

Long-context modeling

Yale University

20

10/16/24

• Inputs to LLMs are limited by context length – the maximum number of tokens the LLM can receive in one forward pass

Word embedding layer

Large

Language

models

predict

...

Positional

encoding

Long-context modeling

Yale University

21

10/16/24

• Q: How to extend LLM context length?
• Idea 1: Interpolate or extrapolate positional encoding of tokens
• Rotational Positional Encoding (RoPE, Ding et al.) - Apply a rotation to query/key word vectors based on relative position

LongRoPE, 2024

Rotation

Efficient attention

Yale University

22

10/16/24

• Attention mechanism has quadratic complexity with respect to sequence length

FlashAttention, NeurIPS 2022

Improving efficiency:

GPU optimization

Improving scalability:

Sub-quadratic attention

Performer Attention, ICLR 2021

Parameter-efficient finetuning

Yale University

23

10/16/24

• Finetuning all LLM parameters for each downstream task is often not efficient
• Idea: Tune a fraction of weights for each task

Low-rank adaptation of LLMs, ICLR 2022

LoRA:

Add adapter weights to LLM which are tuned in a lower-rank space

Quantization

Yale University

24

10/16/24

• Compression techniques that represent LLM weights using lower precision
• 🡪 lower memory footprint, faster inference
• 🡪 reduced performance

Li et al., 2024

Multimodal LLMs

Yale University

25

10/16/24

• Goal: Enable input/output of other modalities in LLMs
• Challenges
• (1) Aligning other modalities into language space of LLM
• (2) Enabling multimodal generation

Yin et al.., 2023

Foundation Models for Biology

Biological foundation models

Yale University

27

10/16/24

• Biological foundation models train on vast amounts of biological & medical data to gain a general understanding of biological and medical concepts
• Adaptable to diverse downstream tasks in biomedical domains

ESM3

(Hayes et al.)

AlphaFold

(Jumper et al.)

Public biomedical datasets

Yale University

28

10/16/24

Meditron medical training data (Chen et al., 2023)

PubMed

Pub C hem

CDC

arXiv

UniProt

LINCs

GTEx

G

Biomedical LLMs

Yale University

29

10/16/24

• Biomedical LLM performance on QA benchmarks improving over time
• Open-access biomedical LLMs still lag behind closed-source models

Performance of medical LLMs on MedQA (Chen et al., 2023)

Single-cell transcriptomics

Yale University

30

10/16/24

Single-cell sequencing workflow

Image: Azenta, 2021

Single-cell atlases

Human Cell Atlas (Regev et al.)

CellxGene (CZI)

scGPT (Cui et al.)

Yale University

31

10/16/24

• GPT-inspired architecture, models sequences of gene expression tokens
• Downstream tasks: cell annotation, batch integration, perturbation response prediction

Cui et al., Nature Methods 2024

Geneformer (Theodoris et al.)

Yale University

32

10/16/24

• Transformer-based model, models rank-ordered gene expression tokens
• Downstream tasks: chromatin and network dynamics prediction, disease modeling

Theodoris et al., Nature 2023

Challenges of large-scale single-cell modeling

Yale University

33

10/16/24

Main challenges:

1. Multimodality of data associated with single-cell gene expression
1. Cell type and tissue annotations, perturbation and disease labels, manuscripts, gene set databases, knowledge graphs
2. Sparsity and noise inherent in gene expression data
3. Batch effects between samples and datasets

Image generated using Dall-E

Benefit of pretraining on text

Yale University

34

10/16/24

• Pretrained language models have shown to outperform task-specific models in NLP tasks
• BioBERT (Lee et al., 2020): outperformed regular BERT and SOTA models

Lee et al., Bioinformatics 2020

Boyack et al., Scientific Data 2020

PubMed literature visualization

Yale University

35

10/16/24

Thank you!

Questions?

Additional Slides

Economic Impact of AI

Yale University

37

10/16/24

Stanford AI Index Report, 2024

Timeline of LLM releases

Yale University

38

10/16/24

2021

GPT-3

2022

2023

2024

Gopher

GPT-4

PaLM 2

Chinchilla

ChatGPT

Claude

OPT

Llama-3

Llama-2

Gemma-2

Mistral

Megatron

NLG

• Timeline of releases for major LLM models since 2021 (non-exhaustive)

LLM Development Pipeline

39

10/16/24

Andrej Karpathy, State of GPT, 2023

Original Transformer architecture

Yale University

40

10/16/24

• LLMs are based on the Transformer architecture (Vaswani et al., 2017)
• Encoder-decoder architecture
• Components:
• Multi-head attention blocks
• Feedforward layers
• Layer Normalization
• Skip connections
• Positional encoding

Vaswani et al., NeurIPS 2017

Direct Preference Optimization

Yale University

41

10/16/24

• RL objective can be optimized using a simpler binary cross-entropy objective
• Key idea
• DPO implicitly defines the reward as a function of the LLM itself
• Directly trains LLM to increase the relative log probability of preferred to
• dispreferred responses

Rafailov et al., Direct Preference Optimization, NeurIPS 2024

Direct Preference Optimization loss objective

Yale University

42

10/16/24

Scaling laws for LLMs

Yale University

43

10/16/24

Scaling laws for Neural Language Models (Kaplan et al., 2020)

LoRA finetuning

Yale University

44

10/16/24

• LoRA: Low-rank adaptation of LLMs
• Idea: Learn a smaller number of parameters to finetune in a lower-rank space

Low-rank adaptation of LLMs, ICLR 2022

Prompt Tuning

Yale University

45

10/16/24

• Idea: learn continuous prompt tokens for each downstream task – modifies text generation by “learning” optimal prompts

Parameter-Efficient Prompt Tuning, 2021

Long-context modeling

Yale University

46

10/16/24

• Q: How to extend context length of LLMs?
• Idea 2: Compress input context to process long sequences
• Infini-attention (Munkhdalai et al.) – compressive memory module for processing long sequences in chunks

Infini-attention, 2024