1 of 15

With large LM�comes large responsibility

AIvidence – Station F – 14.03.23

2 of 15

About me

Louis Abraham

louis.abraham@yahoo.fr

https://louisabraham.github.io/

  • Graduate from École polytechnique (X2015) and ETH Zurich
  • CEO/CTO @ Secrecy https://secrecy.me/
  • CTO @ Gematria Technologies https://gematria.tech/
  • Board member @ Afer https://www.afer.fr/
  • Doctoral student @ Paris 1 Panthéon-Sorbonne

2

3 of 15

What are Large Language Models?

  • Large neural networks based on the Transformer architecture�(Attention Is All You Need, 2017)
  • Models are the combination of a large architecture and a large training set.
    • GPT-3: 175 Billion of parameters – 800 GB
    • Trained using mostly Common Crawl : 570 GB compressed after filtering – 410 billion of tokens
  • ChatGPT is GPT-3 with human feedback from an army of hired contractors, which improved its capabilities for content moderation and coding.
  • GPT models are developed by OpenAI (Microsoft owns 49%)
  • GPT-4 has been announced today and takes images as input
  • Other players are currently releasing LLM

3

4 of 15

Capabilities of ChatGPT (Feb 13 version)

4

5 of 15

Incapabilities of ChatGPT (Feb 13 version)

​

5

6 of 15

Capabilities of ChatGPT (Feb 13 version)

6

7 of 15

OpenAI vs alternatives

  • GPT-3 and ChatGPT are completely closed: code, dataset, weights
  • GPT-J and GPT-NeoX are fully open and created by EleutherAI. GPT-NeoX has 20B of parameters vs GPT-3’s 175B. Trained on the Pile (825GB).
  • BLOOM is also fully open is comparable in size to GPT-3, costed 3M in compute grand from CNRS and GENCI. Training data is not released.
  • LLaMa was released this week

7

8 of 15

LLaMa

  • released last month by Meta
  • weights leaked and are available through torrents
  • trained on public data only
  • comes in different sizes from 7B to 65B
  • smaller models can run on consumer hardware, even Raspberry Pi (10s/tok)
  • it is possible to fine-tune it: see Alpaca (released yesterday) [demo]

8

9 of 15

Risks of LLM

  • For society:
    • LLM can generate large amounts of spam
      • The web could become unsearchable because of SEO optimization
    • Zero-shot ChatGPT passes a lot of exams, including in medicine and law
      • LLM can replace people
      • Jobs and education will evolve
    • LLM make thought poorer by organizing ideas in a standard / average way
  • For AI reliability:
    • LLMs can hallucinate fake but realistic content
    • ChatGPT can be “tortured” into saying anything using prompt engineering
    • ChatGPT is politically left biased
  • For competition?
    • High but not crazy training costs (1-10M)
    • ChatGPT-level quality requires human feedback but other models can generate it cheaply

9

10 of 15

Hallucinations

Latest versions of ChatGPT do much better…

10

11 of 15

Hallucinations

…but it’s far from being perfect

11

12 of 15

LLM try to be politically correct…

12

13 of 15

…but LLM can be “tortured” using prompts

13

14 of 15

It is still hard to detect LLM outputs, even for OpenAI

​

​

14

15 of 15

Thank you!�Questions / Debate

15