CSE 10124�Embeddings 03

Logistics: Exam 01

Where are we?

Today we’ll talk about linear transformations.

​

This is the process through which embeddings are actually created!

Training a Model

Magic Box

SLM

Prompt:

“Hello there”

Tokens:

{“hello”: 3, “there”: 7}

Embeddings:

{3: <0.4, 0.3, …, -0.02>

7: <0.27, 0.8, …, 0.06>}

Predicted Output:

<0.19, 0.14, …, 0.08>

<0.19, 0.14, 0.12, 0.10, 0.09, 0.08, 0.07, 0.06, 0.07, 0.08>

​

0 1 2 3 4 5 6 7 8 9

Predicted Output

Cross Entropy Loss

<0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0>

​

0 1 2 3 4 5 6 7 8 9

Expected Output

= 1.97

Ground Truth:

Hello there general

Deep Learning: Loss Functions

A loss function allows us to numerically measure how far off our model’s predicted values are from what we expected them to be, or what we know the right answer is.

Cross-Entropy: The loss function used to train LLMs is called “Cross-Entropy Loss” and it’s one of the most common loss functions. Heavily based on the principle of “entropy” as used in Information Theory, it’s a measure of how surprising our answer is relative to our ground-truth based on the entropy formula: -∑ p(x)log₂p(x)

Mostly outside the scope of the class, but it’s really not that complicated.

Text Generation

Magic Box

SLM

Prompt:

“Hello there”

Tokens:

{“hello”: 3, “there”: 7}

Embeddings:

{3: <0.4, 0.3, …, -0.02>

7: <0.27, 0.8, …, 0.06>}

Predicted Output:

<0.19, 0.14, …, 0.08>

<0.19, 0.14, 0.12, 0.10, 0.09, 0.08, 0.07, 0.06, 0.07, 0.08>

​

0 1 2 3 4 5 6 7 8 9

Predicted Output

Vocab

{0: “general”, 1: “bill”, 2: “kenobi”, 3: ”hello”, 4: “kitty”,

5: “mom”, 6: “dad”, 7: “there”, 8: “big”, 9: “bill”}

​

Output:

“Hello there” + “general”

Input and Output Embeddings

Magic Box

SLM

Prompt:

“Hello there”

Tokens:

{“hello”: 3}

Embeddings:

{3: <0.4, 0.3, …, -0.02>}

Predicted Output:

<0.19, 0.14, …, 0.08>

<0.19, 0.14, 0.12, 0.10, 0.09, 0.08, 0.07, 0.06, 0.07, 0.08>

​

0 1 2 3 4 5 6 7 8 9

Predicted Output

Embeddings:

3: <1, 1024>

Somehow in here our

embedding changes from:

​

(1, 1024) to (1, 10)

Matrix Multiplications

Linear Transformations

Writing MNIST “Equations”

Visualizations