Conversational System for Differential Diagnosis of GI Cancer

Manjira Sinha, Rajat Pal, Tirthankar Dasgupta

TCS Research and Innovation

December 11, 2024

Human digestive tumors affected by the extract of S. nigrum Linn. and... | Download Scientific Diagram

https://www.gicancersalliance.org/resources/gastrointestinal-cancers-an-urgent-need/

TCS confidential

3

Roles of F‑box proteins in human digestive system tumors (Review)

Overview

TCS confidential

4

TCS confidential

5

• Survival Rates for GI Cancers
• The mortality rates for GI cancers are influenced by how early the cancer is diagnosed, how aggressive it is, and the treatment options available. For example:
• Colorectal Cancer: The 5-year survival rate for colorectal cancer varies by stage. For localized disease (confined to the colon or rectum), the survival rate is over 90%. However, if the cancer has spread to distant organs, the survival rate drops to around 15%.
• Stomach Cancer: The 5-year survival rate for stomach cancer is about 32%, but it is much lower if the cancer is diagnosed in the later stages.
• Liver Cancer: The survival rate for liver cancer is poor, with a 5-year survival rate of about 20%, but this varies based on factors like liver function, stage of cancer, and treatment options.
• Pancreatic Cancer: Pancreatic cancer has one of the lowest survival rates of any cancer, with a 5-year survival rate of around 10%. It is typically diagnosed at an advanced stage.
• Esophageal Cancer: The survival rate for esophageal cancer is also low, with a 5-year survival rate of approximately 20%, but this varies depending on the stage of the disease and treatment methods.

TCS confidential

6

TCS confidential

7

TCS confidential

AMA Augmented Intelligence Research

TCS confidential

Physicians have mixed feelings about AI in Healthcare | athenahealth

Insights 2024: AI' Report - Researchers and Clinicians Believe AI's Potential but Need Transparency

TCS confidential

10

Development of a Question Answering based conversational system that can help in the early detection of GI cancer, given information on general symptoms, diagnosis and medical history of a patient.

TCS confidential

11

TCS confidential

12

30 Question - Answer

TCS confidential

13

50 Questions

TCS confidential

14

Cancer-Answer: Empowering Cancer Care with LLMs

• Achieved maximum A1 score of 0.546 and A2 score of 0.881 across three runs, showing improvement with each iteration.

​

• Prompt engineering provides a flexible, efficient, and scalable solution for building medical QA systems with minimal training data

​

• LLMs like GPT-3.5 Turbo, empowered by prompt engineering, offer a promising tool for improving GI cancer diagnostics and patient care by providing fast access to accurate information. Further research is needed to improve accuracy in highly specialized cases and integrate LLMs into clinical workflows.

• Dataset: 30 GI cancer-related queries for training and 50 for testing.
• Method: Prompt engineering with GPT-3.5 Turbo in zero-shot mode with varying prompts focusing on key idea summarization and direct question answering.

​

• Two metrics were used:

​

• A1 (Entity Accuracy): Fraction of entities present in both the LLM-generated answer and the gold standard answer.
• A2 (Linguistic Correctness & Meaningfulness): Assesses the quality and relevance of the generated response compared to the gold standard.

​

• GPT-3.5 Turbo demonstrates potential for generating accurate and relevant information regarding GI cancers.

​

Values for A1 and A2 across three runs.

• The system effectively interprets complex and unstructured user queries related to GI cancers.

​

​

• Evaluation metrics (precision, recall, F1-score, cosine similarity, Levenshtein distance) demonstrate good performance in terms of accuracy, fluency, and relevance.

​

• Data : A dataset of 186k PubMed articles related to GI cancers was collected and preprocessed to create a robust vector database.
• Retrieval (RoBERTa large: Upon receiving a user query (categorized by cancer type and keywords), the system retrieves the top 50 most similar articles based on cosine similarity, further refined using keyword boosting.
• Generation (BioGPT): The top 10 ranked articles are fed to BioGPT, to generate a comprehensive and informative response to the user's query.
• Model Architecture: A combined system using query categorization, RoBERTa-based retrieval from a vector database, keyword boosting, and BioGPT-based response generation.

​

GI Cancer Diagnostic Chatbot using RoBERTa and RAG

• The input question and text passage were tokenized and fed into the BERT model. The context was divided into segments based on cause-symptoms, symptoms-mutations, and mutations-treatments to accommodate the token limit.

​

• BLEU, ROUGE-1, and ROUGE-2 scores were relatively low. This was attributed to the limited dataset used for training and the token limit in BERT, which resulted in shorter generated answers compared to the reference answers.
• The proposed BERT-based question-answering system demonstrates the potential of using alternative to LLMs to provide accurate and relevant information about GI cancers.

• Dataset: Wikipedia and other public sources (journals, articles, and websites) were used to compile information about eight GI cancers (GIST, esophageal, pancreatic, gall bladder, stomach, liver, anal, and colorectal).

​

• Model Architecture: A BERT large model (bert-large-uncased-whole-word-masking-finetuned-squad) was fine-tuned for question answering.

​

• The process involved simplifying the input question, splitting the context into smaller chunks, parsing each chunk for potential answers, calculating a score for each parsed chunk, and selecting the answer with the highest score.
• Implementation:

​

Gastrointestinal Cancer Related Question Answering Using BERT

Several metrics were used to evaluate the model, including:

• BLEU score
• ROUGE-1, ROUGE-2, ROUGE-L scores
• BERTScore F1
• Average Levenshtein Distance
• Average Cosine Similarity
• Average Perplexity
• Average Relevance Score
• Average Coherence Score
• Precision, Recall, and F1-score for entity extraction

and provide more comprehensive support to clinicians in managing GI cancers.

​

• Corpora Development: Create a conversational AI corpus focused on GI cancer diagnostics using 13,000 structured question-answer pairs from PubMedQA. This dataset covers topics like diagnostics, genetic mutations, disease progression, and treatment strategies.
• The system architecture integrates three primary components:
• BioBERT-NLI: Acts as the encoder for processing user queries and medical text.
• RAG Model: Combines BioBERT-NLI embeddings with a retrieval mechanism (FAISS) and a response generation model (FLAN-T5) to provide contextually relevant and medically accurate responses.
• FLAN-T5: Generates coherent and evidence-backed responses based on information retrieved by the RAG model and user-specific details.

​

Development of a Symptom-Based GI Cancer Diagnostic Bot Using BioBERT-NLI, FLAN-T5 and RAG Model

• dataset : includes comprehensive text-based records from Electronic Health Records (EHRs) related to Gastrointestinal (GI) Cancer. The data collection process involved sourcing relevant medical records, ensuring the inclusion of diverse and representative samples of cancer cases, and focusing on text data that describe symptoms, diagnoses, and patient histories.

​

A Retrieval-Augmented Generation (RAG) Pipeline for GI-Cancer Prediction and Classification Using Quantized Large Language Models

TCS confidential

20

Thank you