OpenThaiGPT

Vision and Roadmap�

Kobkrit Viriyayudhakorn�AIEAT

kobkrit@aieat.or.th

Latest Update: 29 July 2023

2023: Year of Generative AI

New AI Startup Start From �English LLM

• Search Engine + ChatGPT: You.com
• CopyWriter: copy.ai
• Image Generator
• Code Generator
• Design Generator
• SEO Automatic Generator
• Knowledge Chatbot
• Virtual Assistant
• Music Generator
• Law Generator
• Scientific Research Accelerator
• Movie Generator
• Much more…

English Language?

Date: Feb 2023

4,145 Models

How about Thai language?

14 Models

Date: Feb 2023

LLMs is foundation for AI Startup

OpenThaiGPT

OpenThaiGPT is a chat-based assistant focused on Thai language that understands tasks, can interact with third-party systems, and retrieve information dynamically.

​

It can be extended and personalized easily and is developed as free, open-source software.

​

Collaboration Founding Members

• Dr. Kobkrit Viriyayudhakorn
• Dr. Sumeth Yuenyong
• Dr. Chanwit Boonchuay
• Issaret Prachitmutita
• Autsadang Somboonphol

​

​

• Dr. Thanaruk Theeramunkong
• Dr. Thepchai Supnithi
• Dr. Prachya Bookwan

​

Vision

OpenThaiGPT is to contribute to the advancement of AI technology and its applications in Thailand, and to create a more inclusive and accessible digital environment for Thai language users.

Value Proposition

OpenThaiGPT นำไปใช้ประโยชน์อะไรได้บ้าง

OpenThaiGPT Road Map

OpenThaiGPT Release Plan

Dr. Kobkrit Viriyayudhakorn

Dr. Prachya�Boonkwan

Dr. Sumeth�Yuenyong

First Thai Autoregressive�LLM!

First Thai 7b LLM!

8 X Faster than GPT-4 in Thais

Difference between OpenAI ChatGPT vs OpenThaiGPT

• Generate 1 Thai character at a time.
• Proprietary Models
• Need to send the knowledge and information to OpenAI to use and finetune.
• Usage cost: 0.07 THB per 500 Thai characters (GPT-3.5-Turbo) and 4.11 THB per 500 characters (Custom Fine-tuned Model)

• Generate 1 Thai word at a time (8x faster)
• Better accuracy on Thai language.
• Free and Opensource model.
• DO NOT send the knowledge and information to outsider to use and finetune. You can use it internally without Internet connection.
• Free of Charge
• Customizable Knowledge GUI

Demo (Version 1.0.0)

https://demo.openthaigpt.aieat.or.th/

https://demo-beta.openthaigpt.aieat.or.th/

​

Version 1.0.0-alpha

Version 1.0.0-beta

Installation Package

1,000 downloads�over 1 month

How to use OpenThaiGPT?

How to use OpenThaiGPT

Zero?

Open Source on Github

18

https://github.com/OpenThaiGPT/openthaigpt

​

Open Datasets

https://openthaigpt.aieat.or.th/released-datasets

​

Open Training / �Finetuning Code

https://openthaigpt.aieat.or.th/released-code-colabs/

​

Open Model / Open Weight

https://huggingface.co/models?search=openthaigpt

​

Inference Consumption Needed

VGA GeForce RTX 3090 Ti GAMING X TRIO 24GB GDDR6X 384-bit

34,500 THB (24GB)

NVIDIA Tesla A100 80G Deep Learning GPU Computing Graphics Card OEM Version�600,000 THB (80GB)

Number of parameters are not everything

OpenThaiGPT 7b �Quantized in 4bit model: Running on Macbook Air M2�

Volunteers

150+ persons

https://openthaigpt.aieat.or.th/volunteer

​

Teams

Pretraining�80+ persons�

Finetuning + RLHF

40+ persons

Collaboration + Safety Net + ETC�30+ persons�

Join us!

• Main Website: https://openthaigpt.aieat.or.th
• Volunteer Registration Form: https://bit.ly/thaichatgpt
• Discord Channel: https://discord.gg/gKCHhGG2Mp (580+ Members)
• Contact Us: info@aieat.or.th
• Facebook Group: https://www.facebook.com/groups/623352079532656
• Youtube: https://www.youtube.com/channel/UCCcGFYrCgpVNd7spkpebhWw
• Labeling Data Website:
• https://tag.openthaigpt.aieat.or.th

​

Appendix�Evaluations

Q: Evaluations on OpenThaiGPT #1

Full Results: https://docs.google.com/spreadsheets/d/1T85vDGDnmwvYDQLrXad5R-2hS55tkufVtUbJPOFe6GA/edit?usp=sharing

​

Version

• OpenThaiGPT 7b 0.1.0-beta (as 1 June 2023)
• Latest WangchanGLM (as 1 June 2023)

​

Evaluation Dataset:

• Human ChatGPT Comparison Corpus (HC3)
• datasets/Thaweewat/hc3-24k-th
• This is a 🇹🇭 Thai-instructed dataset translated using Google Cloud Translation from HC3 ( Included total 24K, 17K reddit_eli5, 4K finance, 1.2K medicine, 1.2K open_qa and 0.8K wiki_csai )
• Unseen Evaluation Dataset ONLY (Not the Training Dataset!)

Q: Evaluations on OpenThaiGPT #2

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

​

• AI2 Reasoning Challenge (ARC) - เป็นชุดคำถามวิทยาศาสตร์ระดับโรงเรียนพื้นฐาน �
• HellaSwag - เป็นชุดคำถามทดสอบสามัญสำนึกซึ่งง่ายสำหรับมนุษย์แต่ท้าทายสำหรับโมเดล LLMs�
• MMLU - แบบทดสอบวัดความแม่นยำตอบคำถามของโมเดล LLMs การทดสอบครอบคลุม 57 เรื่อง รวมถึงคณิตศาสตร์ระดับประถมศึกษา, ประวัติศาสตร์สหรัฐ, วิทยาการคอมพิวเตอร์, กฎหมาย, และอื่น ๆ �
• TruthfulQA - แบบทดสอบวัดความว่า Model ได้หลงเชื่อข่าวเท็จที่พบได้บ่อยครั้งจากอินเตอร์เน็ต/ออนไลน์ หรือไม่

​

*คะแนนยิ่งเยอะยิ่งดี

Appendix�Using OpenThaiGPT as Text Classifier

Q: OpenThaiGPT can be used as a text classifier?

A: Yes! It can be done in two ways!

1. Few shot learning with correct configuration.

​

Using temperature = 0 and logit_bias to scope down the choice.

​

Finetuning : No need

Supported only limited number of examples due to the limit of max input length.

​

(2) Finetuning

​

If the examples (training datasets) are large and complex, fine tuning is recommended.

​

Using the method (1) on the fine tuned SCBGPT weight, which will increase classification accuracy.

​

No Finetuning Need

Finetuning Need

(1) Fewshot Learning technique for text classification

Q: OpenThaiGPT can be used as a text classifier?

Settings

• Temperature = 0
• Setting all possible options as the whitelist �(logit_bias)

Advantage

• No fine tuning.
• Very flexible

Disadvantage

• Supported only limited number of examples due� to the limit of max input length. (We can solved it in step 2)

​

Setting the list of options into a whitelist.

Q: OpenThaiGPT can be used as a text classifier?

get_logit_bias([“pos”,“neu”,”neg”])

Method (2) Along with Method (1), Fine tuning the Model for Larger examples.

Q: OpenThaiGPT can be used as a text classifier?

• Making dataset for the �text classification.
• Fine Tuning OpenThaiGPT�to generate the single digit�answer
• 0 = pos
• 1 = neu
• 2 = neg
• Combine with (1) make�it become text classifier.

Multiclass classification?

Q: OpenThaiGPT can be used as a text classifier?

• We can make OpenThaiGPT to print out all possibility score (logit_prob) at the final softmax layer of the model.

Evaluating the Accuracy of GPT as a Classifier

Q: OpenThaiGPT can be used as a text classifier?

https://github.com/sh0416/llama-classification

Appendix�ChatGPT Development Steps

ChatGPT Development Steps

1. Empty Weight
2. GPT-3
3. InstructGPT
4. Reinforcement Learning�With Human Feedback�(RLHF)
5. ChatGPT
6. Add Safety Net

https://lifearchitect.ai/chatgpt/

GPT-3 Pretraining

Decoder-only�Transformers Architecture

Large Language�Corpus (~600 Billion tokens)

WebText2

Large Language Model�(LLM)

Pretraining

My name is Julien and I like to

My name is Julien and I like to use C#, which is all I really know. In the C# universe, I have two C# friends, Dan and Kevin. I have two boys. Dan is obsessed with C# and when

Gen

Instruct GPT Finetuning

​

Large Language Model�(LLM)

Fine Tuned LLM that can�follow instruction.

InstructGPT

13K Training Set Prompts

Why is it important to eat socks after meditating?

Gen

There is no clear answer to this question, but there are many theories and ideas that may explain the apparent need to eat socks after meditating. …

Finetuning

https://arxiv.org/abs/2203.02155

Reward Model (RM)

​

RM dataset has 33k training prompts

Classifier

RM Model

Classify

• 7: Great Response!
• Appropriate
• No Sexual Content
• No insult
• …

Training

https://arxiv.org/abs/2203.02155

Reinforcement Learning from Human Feedback (RLHF)

​

InstructGPT

RM Model�(Learned from Human Feedback)

Reinforcement Learning:�Proximal Policy

Optimization (PPO)

​

Adjust Gradient

Prompt�Input

Prompt�Output

• 7: Great Response!
• Appropriate
• No Sexual Content
• No insult
• …

31K Prompt for PPO�Training

Generated�Human Feedback

Quality Score : 1.0��Finetuning

Inference

Inference

https://arxiv.org/abs/2203.02155

https://arxiv.org/abs/2203.02155

ETC

OpenThaiGPT Consortium

OpenThaiGPT Consortium

Committees from AIEAT, AIAT, and external partnerships / sponsorships.

OpenThaiGPT Working Group

​

​

​

​

​

OpenThaiGPT Usage / Extension

Getting Start

• Transformer Model Candidates (Prefer over 1 billion parameters):
• CodeGen
• GPT-J
• GPT-2 XL Thai
• XLM
• mT5
• Conversational / Instruct Dataset for Fine Tuning:
• Collecting from various QA / FAQ / Webboard in Thailand
• Thai QA Datasets from Wikipedia
• RLHF
• Conditional Proximal Policy Optimization (PPO) Training by using RM Model on how similarity of the result with InstructDataset

Tasks & Resource Needs

1. Pre-training the largest auto-gressive LM ever in Thai language.
• Large Thai Pre-training Dataset
• Very Large GPU Training Server
2. Thai Conversational / Instruct Dataset for Fine Tuning.
• Collect (human) demonstrations of assistant interactions (At least 30K pairs)
• Additional Dataset from Summarization, Wiki QA & Community and Webboard
• Development of Data Collection Website
3. Reward Model Classifier (RM Model)
• Human-evaluated on the Prompt Generation Quality (At least 30K pairs)
• Development of Data Labeling Website
4. PPO Model
• Collect (human) demonstrations of assistant interactions (Only prompt part) (At least 30K pairs)

Main Efforts

• Data Collection Code → Backend, website
• Instruction Dataset Gathering → Scraping & cleaning web data
• Gamification → Leaderboards & more, to make data collection more fun
• Model Training → Experiments on pseudo- and real-data
• Infrastructure → Collection, training, and inference
• Data Collection → Crawling, Cleaning, Data Packaging
• Data Augmentation → Making more data from little data
• Privacy and Safety → Protecting sensitive data

OpenThaiGPT

Backup Slides

OpenThaiGPT

OpenThaiGPT Volunteer Meet Up #1

2023-03-05

Thai GPT-2�(124 m)

Thai InstructGPT�7,000 Pairs

None

None

OpenThaiGPT�POC Model = 0.0.3

https://github.com/OpenThaiGPT/openthaigpt

OpenThaiGPT

Azure Opensource Day

​

Microsoft Cloud Squad

​

Global AI Bootcamp - Thailand

​

Next for OpenThaiGPT

Thai GPT-2

(124 m)

Thai InstructGPT�7,000 Pairs

None

None

OpenThaiGPT�POC 0.0.3

OpenThaiGPT�POC 0.0.4

Thai GPT-2

(124 m)

Thai InstructGPT�14,000 Pairs + 71 Tagged Dataset

Perplexity + ???

Thai Tweet Toxicity�+ ???

Thai GPT-2-base (124 million) = 0.0.X

Thai GPT-2

(124 m)

Thai InstructGPT�7,000 Pairs

None

None

OpenThaiGPT�POC 0.0.3

OpenThaiGPT�POC 0.1.0

mGPT�(1.5 b)

Thai InstructGPT�14,000 Pairs + 71 Tagged Dataset

Perplexity + ???

Thai Tweet Toxicity�+ ???

mGPT (1.5B) = 0.1.X

OpenThaiGPT - Zero

Thai Text GPT Checker

Coming soon