1 of 32

Hyperdemocracy

​

Coproducers:

​

HacDC

Singularity DC

Assemble Company

2023-06-23

​

Gabriel Altay

Forrest Murray

Calliope Youngblood

2 of 32

Itinerary

  1. Demo
  2. NLP Primer
  3. Prompt Engineering Primer
  4. Legislation Primer
  5. LUNCH!!!
  6. Tutorial: Code-Along
  7. Break + Special Guest
  8. Choose Your Own Adventure
  9. Project Showcase

​

Extras:

  • Wifi: FREEWIFI-DCPL
  • NO FOOD IN THIS ROOM
  • Keep Drinks Covered

​

3 of 32

4 of 32

5 of 32

Starting with NLP

6 of 32

How similar are these two words?

dog

cat

7 of 32

How similar are these two sentences?

The bank down the street is where I have a savings account.

The river bank is a nice place to have lunch on a sunny day.

8 of 32

How similar are these two vectors?

9 of 32

How similar are these two vectors?

10 of 32

What if you could treat words like vectors?

dog

cat

11 of 32

Embeddings help turn semantics into geometry

12 of 32

Embeddings let you do math with words

Embeddings have linear substructure

​

We have already seen that words with similar meanings will be close to each other in embedding space. In addition, the directions in the vector are related to certain concepts.

​

Analogies can be phrased as math

​

King is to Man as Queen is to Woman

​

becomes

​

King - Man + Woman = Queen

13 of 32

How do you create embeddings?

  1. Find a corpus (e.g. Wikipedia)
  2. Choose a vocabulary (e.g. 50k words)
  3. Choose an embedding size (e.g. 768)
  4. Choose an unsupervised task
    1. predict words in context
    2. predict masked words
    3. predict next word
  5. Train a model to accomplish the task
  6. Use the learned parameters of the model as embeddings

14 of 32

Training Samples (predicting words from context)

15 of 32

Embeddings are initialized randomly and improve as the model trains

16 of 32

17 of 32

Transformer architecture

18 of 32

From lookup tables to attention

  • word2vec produces one embedding per word. for example, the embeddings for “bank” in “bank teller” and “river bank” are the same

​

  • modern architectures like transformers produce “contextualized” embeddings. this means you start with a lookup table and then modify each word’s “base” embedding using information from nearby words.

19 of 32

LLM Application Stew

Generative Language Models

​

e.g. GPT, PaLM

Vector / Doc Stores

​

e.g. Pinecone, Chroma, ES

​

​

Langchain

​

LLM Abstractions

(agents, tools, prompts)

Embedding Models

​

e.g. sentence transformers, data2vec

Structured Indices

​

e.g. LlamaIndex

Data

​

Legislation text/xml/html, co-sponsor graphs,

20 of 32

Prompting is all you need:

LLMs for fun and profit…

21 of 32

Prompt Engineering

22 of 32

How do we use LLMs?

23 of 32

What do LLMs know?

The Good

  • ✅General Knowledge
  • ✅Summarization
  • ✅Small to Medium reasoning

The Bad

  • ⛔Current Events
  • ⛔Private Data

The Sorta

  • 🤷‍♀️Solving specific math and logic problems

24 of 32

How do we fix the bad by using the good? 🤔

  • How do we get LLMs to know about our own data?
  • How do we move from 1 prompt to a conversation?
  • How do we make LLMs better at things they aren’t so good at?
  • How can we get LLMs to do stuff without asking them?

25 of 32

Retrieval Augmented Generation (RAG)

  • parametric memory: information stored in the parameters of the generative seq2seq model

​

  • non-parametric memory: information stored in the dense vector index

26 of 32

🤖✨Magic Prompt

27 of 32

Source: https://www.pinecone.io/learn/langchain-conversational-memory

28 of 32

ChatGPT – Just JSON? Maybe..

29 of 32

🔨🧰🪛Tools

30 of 32

🤖✨Magic Prompt pt. 2

31 of 32

32 of 32

Legal Reasoning