Large Language Models, current state and outlooks

Chaire PARI – 17.10.23

About me

Louis Abraham

louis.abraham@yahoo.fr

https://louisabraham.github.io/

• CEO/CTO @ Secrecy.Tech
• CTO @ Gematria Technologies
• Board member @ Afer
• Doctoral student @ Paris 1 Panthéon-Sorbonne
• Graduate from École polytechnique (X2015) and ETH Zurich

2

What you will learn today

• What are LLM? How are they made? What can they do and not do?
• What are the leading models? Are they open?
• Is is possible to customize LLM?
• What resources are needed to retrain or run LLM?
• How can LLM answer questions using up-to-date information?
• What are applications and current challenges of LLM?
• Will LLM change the world?

What are Large Language Models?

• LLM are large neural networks based on the Transformer architecture�(Attention Is All You Need, 2017) applied to Natural Language Processing

4

The history of LLM

• Since 2018, research has been growing on decoder-only architectures

5

How are LLM made?

• Models are the combination of a large architecture and a large training set.
• GPT-4: 1 Trillion of parameters – 4.5 TB
• GPT-3: 175 Billion of parameters – 800 GB
• Top models are trained using mostly publicly available data
• Training a top performer costs millions (GPT-4 costs $100M)

6

QA capabilities of ChatGPT (Feb 13 version)

7

Coding capabilities of ChatGPT (Feb 13 version)

8

Incapabilities of ChatGPT (Sept 25 version)

​

9

Not better with GPT-4 (Sept 25 version)

What are current pretrained models?

• ChatGPT is GPT with human feedback from an army of hired contractors, which improved its capabilities for content moderation and coding.
• GPT models are developed by OpenAI (Microsoft owns 49%)
• GPT-4 takes images as input
• Other players are releasing LLM: Google released Bart, Meta released LLaMa, LLaMa 2 and Code LLaMa, Mistral (french startup) released their own model this month

11

Are LLM open?

• OpenAI models are completely closed: code, dataset, weights
• Early attempts at open-source models:
• GPT-J and GPT-NeoX are fully open and created by EleutherAI. GPT-NeoX has 20B of parameters vs GPT-3’s 175B. Trained on the Pile (825GB).
• BLOOM is also fully open is comparable in size to GPT-3, costed 3M in compute grant from CNRS and GENCI. Training data is not released.
• Since the release of LLaMa, most players make their models open

12

Recent open-source, open-weight and open-data advances

• LLaMa was released by Meta and weights are public for research
• Meta also released follow-up models Llama 2, Llama Chat and Code Llama
• All LLaMa models are trained using only publicly available data
• Since LLaMa 2, the licence is completely permissive
• A few other organizations followed and released fondation models:
• Falcon released by the Technology Innovation Institute (Abu Dhabi) with open dataset
• Mistral released by Mistral (France) but did not disclose their dataset
• both give no restriction at all on model usage

13

Fine-tuning

• Some models are based on others according to a “fine-tuning” process
• A lot of successful models Alpaca, Vicuna and WizardLM, all finetuned from LLaMa (2)
• Fine-tuning is able to improve further the results of models on specific tasks, like chatting or question answering
• Even complex fine-tuning costs a few hundred dollars compared to a few millions to produce a base model

14

Alignment and uncensoring models

• Most foundational models are censored to prevent the model from generating things like hate speech or adult content
• Alignment gives legal and PR protection to the company releasing models
• But alignment can be removed by documented fine-tuning techniques for a few hundreds dollars

​

A note on model sizes

• Models come in various “sizes” scaling from 7B to 65B parameters. Falcon has 180B parameters.
• These parameters are usually trained using 32-bit or 16-bit floating-point numbers, each parameter taking respectively 4 or 2 bytes.
• After training, models can be “quantized” by representing the coefficients with less bits (typically 4 bits = 0.5 bytes)
• Inference is constrained by the GPU or CPU RAM available
• Quantization makes inference of even 30B models doable on consumer hardware
• Smaller models begin to match the performance of larger models

​

Computational resources

• It is now easy to run open-source state-of-the-art 7B and 13B at high speed (>10 tokens per second) on laptops
• One-liner: `ollama run mistral`
• fine-tuning on large datasets still requires moderate a few dozen GPU hours

Other limitations

• Language: models are mainly trained on english datasets. However, translation models can be used to create a bridge with an english speaking LLM.
• Recent information: incorporating new information into a LLM requires to retrain it but there are other ways to handle it.

​

Handling a knowledge base

• It is possible to give context to a LLM:�You are the chatbot of a company answering to a client. The processing cost of a new subscription is 10 euros. USER: I need a plan for 12 people. How much is the one-time payment?�ASSISTANT: Hello! Thank you for reaching out to our company. For a plan for 12 people, the one-time payment would be 120 euros. This amount includes the processing cost of 10 euros per person. Please let us know if you have any other questions or concerns.

How can we find relevant information?

• With query embeddings and approximate nearest neighbor search

Applications of LLM

• Text generation is very powerful and general
• Chatbots: customer service, question answering
• Personalized marketing
• Text generation can also simply improve the productivity of humans
• But GPT is not just text completion
• Embeddings can be used in regressions or classification, eg underwriting or fraud detection.
• Similarity search can improve question answering with a database
• Specialized classifiers can be put on top to prevent harmful content (as in image models)

21

LLM are the proof that NLP is ready

​

​

• Smaller classifiers like GPT-2 can be much more useful than previous approaches�(eg bag-of-words)
• LLM perform better on most tasks because they possess a form of commonsense knowledge (despite not having commonsense reasoning)

Challenges of LLM

• Reliability
• LLM can hallucinate fake but realistic content
• ChatGPT can be “tortured” into saying anything using prompt engineering
• Alignment is not solved
• Monopolies
• Actually not a problem: training costs from scratch are high but not crazy (1-10M)
• ChatGPT-level quality requires human feedback but other models can generate it cheaply
• Fine-tuning on foundational models is cheap and works great
• Auditability
• LLM are pretrained, often in a way that is not public
• An AI system is now an algorithm but also a training set and an alignment method

Challenges of LLM

• Detection
• LLM can generate large amounts of spam or fake claims
• It is hard to detect LLM outputs, even for OpenAI
• Can LLM be considered legal persons?
• Humans make mistake, probably more than GPT-4
• You cannot open the brain of a human more than a LLM
• Some robots already have citizenship like Sophia in Saudi Arabia
• The European Parliament was favorable to the idea in 2017

Will LLM change the world?

• Good quality LLM are now super cheap to run
• They can automate a lot of question answering and customer support tasks
• These tasks were already being outsourced to countries with cheaper labor when possible
• LLM (coupled with speech recognition and synthesis) are or will get better than low-paid workers
• Are white-collar jobs at risk?
• LLM are enablers, like computers, emails or Microsoft Excel
• White-collar jobs will get more efficient and adapt
• LLM could get authorized to act as agents of persons or organizations

Thank you!�Questions / Debate

26