1 of 24

Drupal4Gov Webinar - April 18, 2024

Implementing LLM Search 4 Gov�Drupal Based Natural Search Chatbot implementation�

2 of 24

2

Welcome
Why LLM and why build an LLM chat search?
Our considerations, approach & strategy
Review architecture
Implementing an AI Chatbot
Demo Time
Q&A

Today’s Agenda

Photo Credit: www.trs.texas.gov

3 of 24

Josh Estep

Sr. Drupal Developer

Andrew Kucharski

President & CEO

MEET YOUR TEAM

Promet Team

Cassey Bowden

Marketing Director

4 of 24

Drupal Experts since 2003!

With experts in specializations across several disciplines, Promet strives to bring knowledge and expertise to our work. Every developer on your project will have a Drupal Certification ensuring best practices. This puts us at top 1% of all Drupal agencies in the world in terms of certifications.

4

5 of 24

We strive for long term clients

6 of 24

6

Why LLM & why build an LLM chat search?

7 of 24

7

https://openai.com/research/gpt-4

8 of 24

8

https://www.anthropic.com/news/claude-3-family

9 of 24

Our Adoption of AI

9

10 of 24

Our Adoption of AI

10

11 of 24

Our Adoption of AI

11

12 of 24

Our Adoption of AI

12

13 of 24

Our Adoption of AI - Drupal.org/project/metatag_ai

13

14 of 24

Why are we interested in LLM search

An LLM-based search, such as one that uses models like ChatGPT, offers several advantages over traditional search engines, providing a more intuitive and user-friendly experience:

Natural Language Understanding: LLMs can handle queries expressed in natural, conversational language. This capability allows users to ask questions just as they would in a normal conversation, reducing the need to craft queries specifically optimized for search engines.
Contextual Responses: Unlike traditional search engines that typically return a list of links, LLMs provide direct answers by synthesizing information from various texts. This approach saves users time by eliminating the need to sift through multiple web pages to find relevant information.
Conversational Interaction: LLMs support ongoing dialogue, allowing users to ask follow-up questions or seek clarifications without losing the thread of the conversation. This continuous interaction makes the search process more dynamic compared to the static outputs of traditional search engines.

14

15 of 24

Martin County Vision

To integrate an LLM supported chatbot conversational search feature into the Martin County Library section of the Drupal website.
To improve the search functionality, making it more user-friendly, interactive, and efficient.
To leverage ChatGPT, or similar LLM model for generating dynamic, context-aware search responses.

15

16 of 24

Our Adoption of AI - AWS Version

https://aws.amazon.com/blogs/big-data/build-scalable-and-serverless-rag-workflows-with-a-vector-engine-for-amazon-opensearch-serverless-and-amazon-bedrock-claude-models/

16

17 of 24

Workflow - How it Works

User Asks Question
Check for violation (not shown)
Ask LLM to analyze question �For keywords
Perform Solr search
Send results and questions�To LLM (RAG)
Respond to user

17

18 of 24

18

Implementation of AI Chatbot

18

19 of 24

Phase I: Planning

Identify

Goals and use-cases; what information and services should it provide?
Chatbot personality and tone.
AI solution or solutions to use; for example OpenAI.

Example questions and ideal answers from stakeholders.

Obtain

Prototyping to verify approach for implementation.

Conduct

19

20 of 24

Phase II: Development

Implement

Queuing mechanism - A queue to handle multiple concurrent user conversations.
Chatbot UI - Accessibility / Device support - Mobile/tablet/desktop
Safeguards

Chat history - Requirements for data retention.
Usage - Check the number of messages to prevent abuse.
Moderation - Ensure user message do not violate LLM API terms-of-use.

Compression - Logic to compress long conversations while maintaining context.

Use a less complex LLM model to handle utility tasks such as compression if needed; this can reduce expenses and ensure faster responses.

Optimize

20

21 of 24

Phase III: Prompt Engineering

Implement Chatbot Behavior

Use the list of example questions/responses to use verify the chatbot is working correctly

This may involve adjustments to the system message as well as "phrase value pairs"
Each time a change is made, test-cases should be re-checked to avoid regressions.

In coordination with stakeholders, iterate on chatbot settings to ensure it passes all test-cases
During this process, additional example questions/answer are likely to be identified.

Refine

21

Role: Josh

Prompt Engineering

Getting the chatbot's behavior right is an interative process.

In coordination with the client, gather a list of example questions/responses to use verify the chatbot is working correctly -- So, if it answers all questions on this correctly then it is considered to be working correctly. Some of these examples might require a full conversation or several message.
Get the system message right

Each LLM conversation starts with a system message the AI sees before the conversation starts that tell it how to behave.

As you test, it's likely you'll identify edge-cases that need to be addressed in the system message:

For example, we added multilingual support simply by mentioning it in the system message.

And then, in coordination with the client, refine the chatbot's settings until it's handles all the example conversations successfully.
Debug mode - One way to facilitate this to add a way to enable a "debug mode" for the chatbot's subsequent response.

To do this, I added code that looks for a particular phrase and then, if it is found, modify the system message so the chatbot will answer freely.

This way, if it's not behaving as expected, you can ask it why?

22 of 24

22

Demonstration

22

23 of 24

23

Strategy and Implementation

24 of 24

24

Q&A

Resource Link: https://www.prometsource.com/drupal4gov2024