1 of 48

​

​

The Rise of

Generative AI

BOTI 2023

1

Farrukh Nauman 

​

RISE Research Institutes of Sweden AB

TITELSIDA

Du byter bakgrund genom att högerklicka på bakgrunden och välja Formatera bakgrund.

Byt stil genom att sätta markören i stycket och använd knapparna för ökat och minskat indrag.

Nivå 1: Titel�Nivå 2: Förtitel�Nivå 3-4: Eftertitlar�

TITLE SLIDE

To change the background, right click on the background and select Format background.

To change the style, put the marker inside the paragraph and use the buttons for indents.

Level 1: Title�Level 2: Pretitle�Level 3-4: Subtitles

2 of 48

RISE

  • ~3500 employees
  • Spread all over Sweden
  • Public Research Organization
  • Headquarters: Borås

BOTI 2023

2

TEXTSIDA

Anpassa huvudtextrutans höjd baserat på rubrikens höjd. Använd stödlinjerna för korrekt placering.�

TEXT SLIDE

Adjust the height of the main text frame based on the height of the heading. Use the guides to find the correct placement.

3 of 48

Center for applied AI

Transport and mobility

Built environment

Digital innovation

Computer�vision

Interaction design

Food and agriculture

Energy systems

Digital ethics

Advanced�data analysis

Deep�learning

Chemistry and pharma

Healthcare

Automated reasoning

Policy innovation

Industrial automation

Climate

change

Knowledge representa-tion

Natural�language

processing

Planning and scheduling

Education

Cyber security

AI agenda

AI �platforms and edge

Software engineering

AI at RISE�(nearly 500 AI researchers/developers)

Materials and production

ICE�data center

ToD:�Cyber Range

Telecom

Technology�areas��Synergistic�areas��Application�areas��Test & Demo��Special�projects

ToD:�5G/Edge

ToD:�AstaZero

4 of 48

Hot off the press

  • LLMs: Two more GPT-4 class models �Claude-Opus�Gemini-Ultra
  • 3D Vision and Video models�OpenAI Sora�Stable Diffusion Video
  • Others �Hardware improvements: Intel, AMD, …�Agents: Devlin (Autonomous software engineer),

BOTI 2023

4

TEXTSIDA

Anpassa huvudtextrutans höjd baserat på rubrikens höjd. Använd stödlinjerna för korrekt placering.�

TEXT SLIDE

Adjust the height of the main text frame based on the height of the heading. Use the guides to find the correct placement.

5 of 48

Agenda

  • Overview: AI, Why now?
  • Large Language Models: Training, Evaluations, RAG
  • Multi-modal Models
  • Societal Impact

BOTI 2023

5

TEXTSIDA

Anpassa huvudtextrutans höjd baserat på rubrikens höjd. Använd stödlinjerna för korrekt placering.�

TEXT SLIDE

Adjust the height of the main text frame based on the height of the heading. Use the guides to find the correct placement.

6 of 48

AI Overview

BOTI 2023

6

TITELSIDA

Du byter bakgrund genom att högerklicka på bakgrunden och välja Formatera bakgrund.

Byt stil genom att sätta markören i stycket och använd knapparna för ökat och minskat indrag.

Nivå 1: Titel�Nivå 2: Förtitel�Nivå 3-4: Eftertitlar�

TITLE SLIDE

To change the background, right click on the background and select Format background.

To change the style, put the marker inside the paragraph and use the buttons for indents.

Level 1: Title�Level 2: Pretitle�Level 3-4: Subtitles

7 of 48

What is AI?

BOTI 2023

7

Stuart J. Russell and Peter Norvig, "Artificial Intelligence A Modern Approach (Third Edition)", Pearson, 2010.

ISO/IEC 22989:2022 Information technology — Artificial intelligence — Artificial intelligence concepts and terminology

AI

<discipline> research and development of mechanisms and applications of AI systems

​

AI System

engineered system that generates outputs such as content, forecasts, recommendations or decisions for a given set of human-defined objectives

Thinking Humanly:

The cognitive modelling approach

Thinking Rationally:

The “laws of thought” approach

Acting Humanly:

The Turing Test approach

Acting Rationally:

The rational agent approach

8 of 48

What is AI?

BOTI 2023

8

AI

ML

Symbolic

AI

Deep Learning

Hybrid

AI

Expert Systems

Statistical Learning

Neural Symbolic Systems

Logic

Illustration Inspired by:

Digitalized Transformation Guides vol. 2, Loughborough University

Jerod Santo, “Disentangling AI, machine learning, and deep learning”, Changelog news, 2021-04-29

Calegari, R., Ciatto, G., Denti, E., & Omicini, A. (2020). Logic-based technologies for intelligent systems: State of the art and perspectives. Information, 11(3), 167.

9 of 48

BOTI 2023

9

Kaiming He, Deep Learning Bootcamp, MIT Schwarzman College of Computing

10 of 48

BOTI 2023

10

Kaiming He, Deep Learning Bootcamp, MIT Schwarzman College of Computing

11 of 48

Why now?

BOTI 2023

11

Model architecture

Data

Computing power

TEXTSIDA

Anpassa huvudtextrutans höjd baserat på rubrikens höjd. Använd stödlinjerna för korrekt placering.�

TEXT SLIDE

Adjust the height of the main text frame based on the height of the heading. Use the guides to find the correct placement.

12 of 48

Large Language Models

BOTI 2023

12

TITELSIDA

Du byter bakgrund genom att högerklicka på bakgrunden och välja Formatera bakgrund.

Byt stil genom att sätta markören i stycket och använd knapparna för ökat och minskat indrag.

Nivå 1: Titel�Nivå 2: Förtitel�Nivå 3-4: Eftertitlar�

TITLE SLIDE

To change the background, right click on the background and select Format background.

To change the style, put the marker inside the paragraph and use the buttons for indents.

Level 1: Title�Level 2: Pretitle�Level 3-4: Subtitles

13 of 48

Best LLMs (March 13th, 2024)

BOTI 2023

13

https://chat.lmsys.org/

14 of 48

Best LLMs (Hallucinations)

BOTI 2023

14

https://huggingface.co/spaces/vectara/leaderboard

15 of 48

Large Language Models���Training��Scaling��Enhancements: RAG, Agents��Evaluations

BOTI 2023

15

16 of 48

Phase 1

BOTI 2023

16

Jay Alammar, How GPT3 Works - Visualizations and Animations

​

17 of 48

Phase 2: Instruction finetuning

BOTI 2023

17

Hyung Won Chung, Large Language Models in 2023.

18 of 48

Why Phase 3: RLHF?

BOTI 2023

18

1. Pretraining: Predict the next token, minimize perplexity.

​

2. Instruction finetuning: Given an input instruction, predict the single correct response. ("behavior cloning" in Reinforcement Learning).

​

Problem: How do we get answers to more complex questions?

Hyung Won Chung, Large Language Models in 2023.

19 of 48

Reinforcement Learning

BOTI 2023

19

Hyung Won Chung, Large Language Models in 2023.

20 of 48

Annotations

BOTI 2023

20

Weng, Lilian. (Feb 2024). “Thinking about High-Quality Human Data”. Lil’Log.

Aroyo, L., & Welty, C. (2015). Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation. AI Magazine, 36(1), 15-24.

21 of 48

Human preferences

BOTI 2023

21

“Training language models to follow instructions with human feedback”, Oyang et al.,  2022; arXiv:2203.02155.

22 of 48

BOTI 2023

22

Hyung Won Chung, Large Language Models in 2023.

Using human preference annotations

23 of 48

Scaling

BOTI 2023

23

Sasha Rush, LLMs in 5 formulas

24 of 48

Scaling

BOTI 2023

24

Sasha Rush, LLMs in 5 formulas

Overparametrized

25 of 48

Emergence

BOTI 2023

25

  1. Unpredictability: Transition at unexpected scales.
  2. Sharpness: Sudden jump in performance.

Are Emergent Abilities of Large Language Models a Mirage?, R. Schaeffer, B. Miranda, Sanmi Koyejo, NeurIPS Outstanding Paper 2023

​

26 of 48

Emergence

BOTI 2023

26

Are Emergent Abilities of Large Language Models a Mirage?, R. Schaeffer, B. Miranda, Sanmi Koyejo, NeurIPS Outstanding Paper 2023

​

27 of 48

Limitations

  • Inference speed (Solutions: speculative decoding, mixture of experts, …)�
  • Finetuning too expensive (Solutions: LoRA – Low ranked quantization that does not change base weights)�
  • Exposure of private data (Solutions: Use a locally hosted small LM; Azure GPT-4 custom finetunes)�
  • High compute requirements (Solutions: Use (expensive) APIs – no need for local compute; Small LMs, New hardware improvements – hard coded transformer weights, neuromorphic computing, Intel Gaudi2, …)�
  • Hallucination and data duplication risks (Solutions: Better evaluations and finetunes)

BOTI 2023

27

TEXTSIDA

Anpassa huvudtextrutans höjd baserat på rubrikens höjd. Använd stödlinjerna för korrekt placering.�

TEXT SLIDE

Adjust the height of the main text frame based on the height of the heading. Use the guides to find the correct placement.

28 of 48

Limitations

BOTI 2023

28

Yann LeCun, Do large language models need sensory grounding for meaning and understanding?

TEXTSIDA

Anpassa huvudtextrutans höjd baserat på rubrikens höjd. Använd stödlinjerna för korrekt placering.�

TEXT SLIDE

Adjust the height of the main text frame based on the height of the heading. Use the guides to find the correct placement.

29 of 48

Retrieval Augmented Generation

BOTI 2023

29

Akari Asai, https://acl2023-retrieval-lm.github.io/

30 of 48

Retrieval Augmented Generation

BOTI 2023

30

Akari Asai, https://acl2023-retrieval-lm.github.io/

31 of 48

Retrieval Augmented Generation

BOTI 2023

31

Akari Asai, https://acl2023-retrieval-lm.github.io/

32 of 48

Retrieval Augmented Generation

BOTI 2023

32

Akari Asai, https://acl2023-retrieval-lm.github.io/

33 of 48

BOTI 2023

33

Akari Asai, https://acl2023-retrieval-lm.github.io/

34 of 48

Agents

​

​

​

BOTI 2023

34

Lilian Weng, Open AI

Information retrieval

Translation across languages and frameworks

Improve productivity

Planning

Code and content creation

Chain of thought and self reflection

TEXTSIDA

Anpassa huvudtextrutans höjd baserat på rubrikens höjd. Använd stödlinjerna för korrekt placering.�

TEXT SLIDE

Adjust the height of the main text frame based on the height of the heading. Use the guides to find the correct placement.

35 of 48

Evaluations

BOTI 2023

35

Open-source benchmarks not reliable due to data leakage.��GPT-4/Opus as judge

​

Gold standard dataset (specific to domain; not shared publicly)

​

Arenas are generally more reliable:�Chatbot Arena�RAG Arena�VLM Arena�

36 of 48

BOTI 2023

36

37 of 48

Multi-Modal Models

BOTI 2023

37

TITELSIDA

Du byter bakgrund genom att högerklicka på bakgrunden och välja Formatera bakgrund.

Byt stil genom att sätta markören i stycket och använd knapparna för ökat och minskat indrag.

Nivå 1: Titel�Nivå 2: Förtitel�Nivå 3-4: Eftertitlar�

TITLE SLIDE

To change the background, right click on the background and select Format background.

To change the style, put the marker inside the paragraph and use the buttons for indents.

Level 1: Title�Level 2: Pretitle�Level 3-4: Subtitles

38 of 48

Multi-modal models

BOTI 2023

38

https://huggingface.co/spaces/WildVision/vision-arena

39 of 48

CAD

BOTI 2023

39

Generating realistic 3D objects using a text prompt – useful for 3D printing, 3D design, etc.

TheGradient Blog: text to cad

40 of 48

3D and Video

BOTI 2023

40

Input: �Text�Single image

​

Novel view synthesis

​

​

41 of 48

3D and Video

BOTI 2023

41

42 of 48

Societal Impact

BOTI 2023

42

TITELSIDA

Du byter bakgrund genom att högerklicka på bakgrunden och välja Formatera bakgrund.

Byt stil genom att sätta markören i stycket och använd knapparna för ökat och minskat indrag.

Nivå 1: Titel�Nivå 2: Förtitel�Nivå 3-4: Eftertitlar�

TITLE SLIDE

To change the background, right click on the background and select Format background.

To change the style, put the marker inside the paragraph and use the buttons for indents.

Level 1: Title�Level 2: Pretitle�Level 3-4: Subtitles

43 of 48

BOTI 2023

43

44 of 48

Concerns

BOTI 2023

44

45 of 48

Concerns

BOTI 2023

45

46 of 48

Risk mitigation

BOTI 2023

46

47 of 48

Conclusions

  • Generative AI is exciting and enables innovation.�
  • Challenges: �1. Deployment (speed, local vs cloud hosting).�2. Risks (privacy, data duplication, …)�
  • A two edged-sword

BOTI 2023

47

TEXTSIDA

Anpassa huvudtextrutans höjd baserat på rubrikens höjd. Använd stödlinjerna för korrekt placering.�

TEXT SLIDE

Adjust the height of the main text frame based on the height of the heading. Use the guides to find the correct placement.

48 of 48

Farrukh

Nauman

Researcher in�Generative AI

farrukh.nauman@ri.se

BOTI 2023

48

KONTAKTER

Byt textnivå genom att sätta markören i stycket och använd knapparna för ökat och minskat indrag.

Nivå 1: Namn�Nivå 2: Titel/info�

CONTACTS

To change the text level, put the marker inside the paragraph and use the buttons for indents.

Level 1: Name�Level 2: Title/info