1 of 15

πŸΌπŸ¦™ cria

Reflections and Learnings from

crafting a LLM app prototype

18 Aug 2023

CRIA v1.3

🌐 chat.walterteng.com

​

πŸ€— davzoku/cria-llama2-7b-v1.3

davzoku/cria

2 of 15

Contents

What is CRIA?

01.

Guiding questions and decisions on developing CRIA

Considerations

02.

Hint: krΔ“-Ι™; plural crias. : a baby llama

3 of 15

What is CRIA?

NousResearch/

Llama-2-7b-chat-hf

Base Model

mlabonne/CodeLlama-2-20k

Instruction Dataset

Custom FastAPI

API Server

Bespoke Next.js + Chakra UI Web App

User Interface

  • β€œCrafting a Rapid prototype of an Intelligent llm App using open source resources”

​

  • an LLM app prototype that demonstrates the integration of the following 4* components:

πŸ¦™

πŸ—„οΈ

🌐

πŸ’»

* cloud deployment is the 5th component; to be explored as part of future work

4 of 15

Base Model Considerations

  • Performance
    • See πŸ€— Open LLM Leaderboard
    • Read the paper / docs; and about the creators

​

  • Size
    • 7B or 70B parameters??
    • Keep in mind of downstream events; eg. deployment and inference hardware requirements

​

  • Pre-Training dataset used
    • Eg. multilingual?

​

  • Base Model or Instruction-Tuned Model?

​

  • License

5 of 15

Dataset Considerations

  • How to Generate?
    • Human-written? How to scale cheaply?
    • LLM-Generated? How to prompt? Eg. Self-Instruct, Evol-Instruct, Orca etc.

​

  • Quality vs Quantity
    • LIMA shows that 1000 high quality instructions is sufficient

​

  • Prompt Style
    • Adhere to base model’s style if applicable; check model card
    • Leverage on existing open source dataset by converting it to the desired instruct format for your use case

​

​

​

6 of 15

Prompt Style Example

7 of 15

Prompt Style Example

8 of 15

Training Considerations

  • Where to Run Training?
    • Local or Cloud?

​

  • What Resources is needed? What is the cost? How long does the training take?
    • Philipp Schmid showed that training a Llama-7B (with Flash Attention) on AWS g5.2xlarge for 2:08 hr, using Dolly-15k dataset costs $2.60
      • g5.2xlarge is EC2 instance with 8 vCPU, 32GB Memory, and 450GB SSD
    • Alternatively, we can training on a free colab instance for <1 hr with 1k dataset

​

​

9 of 15

Colab

Notebooks Demo

​

Part 1, Part 2

10 of 15

API Server Considerations

  • Write an FASTAPI script from scratch?
    • Full control on the API interface
    • More lean, less bloat
    • Need to expose the LLM parameters from scratch

​

  • Utilize OpenLLM?
    • Supports popular open-source model from scratch
    • Supports PEFT-compatible adapters
    • Able to add custom model to the ecosystem
    • No need to reinvent the wheel
    • Supports other goodies such as LangChain and HuggingFace transformers agents integration
    • SSE not supported

​

  • Utilize HF Text Generation Inference?
    • to be explored in the future *

​

​

11 of 15

Front End Considerations

  • Do-It-All in Python?
    • Use Streamlit, Gradio, or text-generation-ui
    • Get fully functional front end with no fuss
    • Easy to maintain for Python devs

​

  • Use Javascript Frameworks Such as Next.JS?
    • More versatile and customizable to integrate a visually appealing design system
    • Supports other goodies such as SEO, PWA
    • Easier integration with existing websites built with JS frameworks

​

​

12 of 15

Cloud Deployment Considerations

  • What Resources is needed? What is the cost?

From Deploy Llama 2 7B/13B/70B on Amazon SageMaker

From Amazon SageMaker Pricing

13 of 15

Preliminary Model Evaluation

From πŸ€— Open LLM Leaderboard

As of 18 Aug 2023

14 of 15

CRIA Demo

​

chat.walterteng.com

15 of 15

Thank You

For

Your Attention

πŸΌπŸ¦™

​