1 of 24

Drupal4Gov Webinar - April 18, 2024

Implementing LLM Search 4 Gov�Drupal Based Natural Search Chatbot implementation

2 of 24

2

  • Welcome
  • Why LLM and why build an LLM chat search?
  • Our considerations, approach & strategy
  • Review architecture
  • Implementing an AI Chatbot
  • Demo Time
  • Q&A

Today’s Agenda

Photo Credit: www.trs.texas.gov

3 of 24

Josh Estep

Sr. Drupal Developer

Andrew Kucharski

President & CEO

MEET YOUR TEAM

Promet Team

Cassey Bowden

Marketing Director

4 of 24

Drupal Experts since 2003!

With experts in specializations across several disciplines, Promet strives to bring knowledge and expertise to our work. Every developer on your project will have a Drupal Certification ensuring best practices. This puts us at top 1% of all Drupal agencies in the world in terms of certifications.

4

©2020 Promet Source. All rights reserved.

4

5 of 24

We strive for long term clients

6 of 24

6

6

6

Why LLM & why build an LLM chat search?

© 2024 Promet Source. All rights reserved

7 of 24

7

© 2024 Promet Source. All rights reserved

8 of 24

8

© 2024 Promet Source. All rights reserved

9 of 24

Our Adoption of AI

9

©2020 Promet Source. All rights reserved.

9

10 of 24

Our Adoption of AI

10

©2020 Promet Source. All rights reserved.

10

11 of 24

Our Adoption of AI

11

©2020 Promet Source. All rights reserved.

11

12 of 24

Our Adoption of AI

12

©2020 Promet Source. All rights reserved.

12

13 of 24

Our Adoption of AI - Drupal.org/project/metatag_ai

13

©2020 Promet Source. All rights reserved.

13

14 of 24

Why are we interested in LLM search

An LLM-based search, such as one that uses models like ChatGPT, offers several advantages over traditional search engines, providing a more intuitive and user-friendly experience:

  1. Natural Language Understanding: LLMs can handle queries expressed in natural, conversational language. This capability allows users to ask questions just as they would in a normal conversation, reducing the need to craft queries specifically optimized for search engines.
  2. Contextual Responses: Unlike traditional search engines that typically return a list of links, LLMs provide direct answers by synthesizing information from various texts. This approach saves users time by eliminating the need to sift through multiple web pages to find relevant information.
  3. Conversational Interaction: LLMs support ongoing dialogue, allowing users to ask follow-up questions or seek clarifications without losing the thread of the conversation. This continuous interaction makes the search process more dynamic compared to the static outputs of traditional search engines.

14

©2020 Promet Source. All rights reserved.

14

15 of 24

Martin County Vision

  • To integrate an LLM supported chatbot conversational search feature into the Martin County Library section of the Drupal website.
  • To improve the search functionality, making it more user-friendly, interactive, and efficient.
  • To leverage ChatGPT, or similar LLM model for generating dynamic, context-aware search responses.

15

©2020 Promet Source. All rights reserved.

15

16 of 24

Our Adoption of AI - AWS Version

16

©2020 Promet Source. All rights reserved.

16

17 of 24

Workflow - How it Works

  • User Asks Question
  • Check for violation (not shown)
  • Ask LLM to analyze question �For keywords
  • Perform Solr search
  • Send results and questions�To LLM (RAG)
  • Respond to user

17

©2020 Promet Source. All rights reserved.

17

18 of 24

18

18

18

Implementation of AI Chatbot

© 2024 Promet Source. All rights reserved

18

19 of 24

Phase I: Planning

Identify

  • Goals and use-cases; what information and services should it provide?
  • Chatbot personality and tone.
  • AI solution or solutions to use; for example OpenAI.
  • Example questions and ideal answers from stakeholders.

Obtain

  • Prototyping to verify approach for implementation.

Conduct

19

20 of 24

Phase II: Development

Implement

  • Queuing mechanism - A queue to handle multiple concurrent user conversations.
  • Chatbot UI - Accessibility / Device support - Mobile/tablet/desktop
  • Safeguards
    • Chat history - Requirements for data retention.
    • Usage - Check the number of messages to prevent abuse.
    • Moderation - Ensure user message do not violate LLM API terms-of-use.
  • Compression - Logic to compress long conversations while maintaining context.

  • Use a less complex LLM model to handle utility tasks such as compression if needed; this can reduce expenses and ensure faster responses.

Optimize

20

21 of 24

Phase III: Prompt Engineering

Implement Chatbot Behavior

  • Use the list of example questions/responses to use verify the chatbot is working correctly
    • This may involve adjustments to the system message as well as "phrase value pairs"
    • Each time a change is made, test-cases should be re-checked to avoid regressions.

  • In coordination with stakeholders, iterate on chatbot settings to ensure it passes all test-cases
  • During this process, additional example questions/answer are likely to be identified.

Refine

21

22 of 24

22

22

22

Demonstration

© 2024 Promet Source. All rights reserved

22

23 of 24

23

23

23

Strategy and Implementation

© 2024 Promet Source. All rights reserved

24 of 24

24

24

24

Q&A

© 2024 Promet Source. All rights reserved