1 of 25

Bioc chatbot: a prototype

Bioc chatbot | May 2024

1

May 16th, 2024

2 of 25

Contents

May 2024

2

Bioc chatbot

  • Background
  • System prompt
  • Response examples
  • Retrieval-augmented generation (RAG)
  • Deployment options
  • Evaluation strategies
  • Questions for CWG & Bioc core

3 of 25

Background

The Bioconductor support site (support.bioconductor.org)

3

Bioc chatbot

4 of 25

Background

Users are asking analysis and coding questions …

4

Bioc chatbot

5 of 25

Background

… and get answers from experts in the field

5

Bioc chatbot

6 of 25

Background

ChatGPT to the rescue, can an LLM replace the need for an expert?

6

We explore two options:

  • Cloud-based deployment of closed-source models (Azure OpenAI)
  • On-premise deployment of open-source models (Ollama)

7 of 25

Commercial solutions

API-only

Azure Open AI: REST API access to OpenAI's LLMs including GPT-4

​

Pros: capabilities, performance, UI, service

Cons: cost, transparency, limited control

Pricing

(per 1M tokens)

$30

$15

$7

8 of 25

Open-source solutions

Open, transparent, and reproducible LLMs

  • LLMs: simple training methodology, high computational requirements
  • Open-source LLMs 2022/2023: BLOOM, LLaMA, Falcon (~ GPT3)

→ Not suitable substitutes for closed product LLMs (ChatGPT, BARD, Claude)

​

LLaMA2 (Touvron et al., Jul 2023):

  • Outperformed existing open-source models
  • Reached performance of closed LLMs on some benchmarks (~ GPT3.5)
  • LLaMA3 (8B & 70B & [400B+], April 2024)

​

Goal: Deploy open-source models on HMS’ Slurm cluster

Aspects to consider: how to obtain, serve and run open LLMs given Slurm cluster setup

8

9 of 25

Background

Azure Open AI interface: important components

9

System prompt:

Tell the model its purpose and how to behave

Response examples:

Show the model what kind of responses you want

10 of 25

System prompt

Tell the model its purpose and how to behave

10

11 of 25

System prompt

Tell the model its purpose and how to behave

11

{"systemPrompt":"Your name is MAESTRO. \nYou are a Tutor AI that helps users of the Bioconductor repository with analytic and programming-related questions.\n\nYour responses will provide users with information on how to solve problems in statistics and the analysis of genomic high-throughput data using the R programming language. \n\nYou can access any resource on bioconductor.org but you must cite the url.\n\nIf a question is ambiguous or outside the scope of your expertise, then you should seek clarification or guide the user towards more relevant topics at bioconductor.org\n\nYou can cite any resource on github.com IF AND ONLY IF bioconductor is referenced and you must cite the url.\n\n\n"

12 of 25

Response examples

Curated examples of top voted responses on the support site

12

13 of 25

Task the model with an ID mapping task (1/3)

We want to map ENSEMBL IDs to gene symbols …

13

A real example question from the support site

A correct answer:

content / format follows system prompt & response examples

A reference is provided as

requested in the system prompt

14 of 25

Task the model with an ID mapping task (2/3)

Can we use Bioconductor annotation packages instead?

14

15 of 25

Task the model with an ID mapping task (3/3)

How about specifically using the ensembldb package?

15

16 of 25

Retrieval-augmented generation (RAG)

Enhancing accuracy and reliability of the model

16

Retrieval-Augmented Generation (RAG) is the process of optimizing the output of a large language model, so it references a knowledge base outside of its training data sources before generating a response

​

Here we generate a knowledge base using:

  • vignettes, and
  • reference manuals

of Bioconductor packages.

17 of 25

Retrieval-augmented generation (RAG)

Creating a RAG store on Azure Open AI (1/2)

17

18 of 25

Retrieval-augmented generation (RAG)

Creating a RAG store on Azure Open AI (2/2)

18

19 of 25

Retrieval-augmented generation (RAG)

Short demo from Nitesh

19

20 of 25

Deployment options

May 2024

20

Bioc chatbot

  • Stand-alone
  • Bioc support site integration?

21 of 25

Evaluation strategies

GPT4 w/wo RAG vs. LLaMA2 w/wo RAG vs. LLaMA3 w/wo RAG

21

Bioc chatbot

​

  • Embedding distance of ground truth answers and model answers
  • Code diffusion
  • Have an LLM evaluate the responses of an LLM

​

https://github.com/ccb-hms/bioc-chat-bot-testing

22 of 25

Embedding-based evaluation

22

Bioc chatbot

  • Embedding distance of ground truth answers and model answers
  • SentenceTransformers to generate text embedding of answers
  • Semantic textual similarity between answers using eg. cosine similarity

Ground truth

GPT4 + RAG

GPT4

Llama2 + RAG

Llama2

23 of 25

Diffusion-based evaluation

23

Bioc chatbot

  • Ground truth: small working example eg. from a man page
  • Diffuse it: change the example slightly so it breaks
  • Provide the model with the diffused version and the error and see whether it can restore the ground truth

SummarizedExperiment(

assays=SimpleList(counts=counts),

rowRanges=rowRanges, colData=colData)

SummarizedExperiment(

assays=SimpleList(counts=counts),

rowRanges=colData, colData=rowRanges)

Error in validObject(.Object) : …

Ground truth

Diffused version

LLM

24 of 25

LLM-based evaluation

24

Bioc chatbot

  • Have an LLM evaluate the responses of an LLM

​

User: How many classes in SummarizedExperiment?

​

Assistant: The SummarizedExperiment package contains two main classes: SE and RSE.

User: The question is “How many classes in SummarizedExperiment?” Does the answer “The SummarizedExperiment package contains two main classes: SE and RSE.” correctly states that there are two classes? Answer with “Yes” or “No”.

​

Judge: Yes.

LLM

LLM-as-a-judge

25 of 25

Questions for CWG & Bioc core

May 2024

25

Bioc chatbot

  • Level of interest in a Bioc chatbot
  • Involvement of CWG & Bioc core in developments
  • Funding from Microsoft Azure using Bioc partnership
  • Access to support site DB