1 of 133

Lavanya.ai

AI • Growth • Startups • Building

PROMPT ENGINEERING

2 of 133

Prompt Engineering

GenAI Enterprise Accelerator Master Class

LAVANYA.AI

3 of 133

Industry leading partnerships

Streamlined End-to-End ML & LLM Workflow

Technology

AI Developer Platform

Customers

1000+ Enterprises & AI Leaders

Leadership

Community & Ecosystem

Ecosystem >50,000 repo & framework integrations

Investors

Weights & Biases

Mission: Build the best tools for AI developers

Series C: $250m in funding

W&B Models

W&B Weave

W&B Core

4 of 133

Are you using GenAI today personally?

LAVANYA.AI

5 of 133

Are you exploring GenAI as a company?

LAVANYA.AI

6 of 133

Are you using GenAI in production?

LAVANYA.AI

7 of 133

Prototype To Production

Nov 4

Foundations of Prompting, RAG systems, Agents

Nov 25

Evaluate and Productionize AI Systems

Dec 16

Build and deploy a GenAI app end-to-end

LAVANYA.AI

8 of 133

Community

8

wandb.me/discord

LAVANYA.AI

9 of 133

Ask Qs!

???

9

LAVANYA.AI

10 of 133

GenAI Accelerator was transformational for our team. Weights & Biases' depth of knowledge on GenAI is unmatched—it's clear why nearly every foundation model is built on their platform. Highly recommended for anyone serious about leveraging GenAI in their business.”

We will help you train your teams on GenAI best practices

accelerator.cc/class

James Cham

Partner, Bloomberg Beta

11 of 133

We will help you build production quality applications

The support we’ve received from Weights & Biases has been exceptional. They have been invaluable as thought partners and guides throughout our journey. Their deep expertise not only allowed us to anticipate potential roadblocks well in advance but also equipped us with well-considered solutions. Thanks to their guidance, we were able to bring the product to market more swiftly and with fewer challenges.”

accelerator.cc

Weave

Deliver GenAI with confidence

Monitor

Collaborate

Evaluate

Build

David Rogier

CEO, MasterClass

12 of 133

Prompt Engineering

1

Agenda

PE – Play

2

RAG

3

RAG – Play

4

Feedback

5

LAVANYA.AI

13 of 133

AI

Machine Learning

Deep Learning

LLMs

GenAI

Foundation Models

LAVANYA.AI

14 of 133

What’s so exciting?

LAVANYA.AI

15 of 133

LLMs massively expedite the ability for anyone to quickly build useful applications

LAVANYA.AI

incredibly performant out of the box
bring down the pace of iteration from months to hours

This is because they can be so incredibly performant out of the box, with just a little bit of prompt engineering. They bring down the pace of iteration from months to hours!

With traditional ML, the steps for putting a useful model in production (getting data, training the model and deploying) took months. With LLMs, you can write good prompts and iterate to find one that works for you in a matter of minutes-hours. You can then start making API calls to the model and making inferences in a couple hours! Not needing massive amounts of data to learn new concepts, and being able to tune your prompts on a small number of examples to get massive lifts in performance is a gamer changer too!

16 of 133

Getting great results from LLMs doesn’t require knowledge of deep learning models, programming, math or statistics.

LAVANYA.AI

17 of 133

Democratization of AI

Foundation

Model Builder

ML

Engineer

Software

Developer

1000s

100,000s

10,000,000s

LAVANYA.AI

18 of 133

Your IP is all the things you tried along the way

LAVANYA.AI

19 of 133

These models are like zip files

compressing the knowledge of the entire internet

LAVANYA.AI

20 of 133

Transformers 🤿

LAVANYA.AI

21 of 133

2017

LAVANYA.AI

22 of 133

Self Attention

LAVANYA.AI

23 of 133

Multi-Headed Attention

LAVANYA.AI

24 of 133

Tokenization

LAVANYA.AI

25 of 133

Tokenizers

25

wandb.me/tokenizer

LAVANYA.AI

26 of 133

How much information can be included in a prompt depends on the model’s context length.

Context is memory

CONTEXT WINDOW

LENGTH

TOKEN

LAVANYA.AI

Context window of an LLM

is the maximum number of tokens
it can consider at once
when generating text

Token length refers to the length of the input, measured in tokens.

How much information can be included in a prompt depends on the model’s context length limit. Models’ maximum context length has increased rapidly in recent years. The first three generations of GPTs have 1K, 2K, and 4K context length respectively. This is barely long enough for a college essay and too short for most legal documents or research papers.

Context length expansion soon became a race among model providers and practitioners. Figure 5-3 shows how quickly the context length limit is expanding. Within 5 years, it grew 2,000 times from GPT-2’s 1K context length to Gemini-1.5 Pro’s 2M context length. 100K context length can fit a moderate-sized book. As a reference, this book contains approximately 120,000 words or 160,000 tokens. 2M context length can fit approximately 2,000 Wikipedia pages and a reasonably complex codebase such as PyTorch.

27 of 133

How much information can be included in a prompt depends on the model’s context length.

Context is memory

LAVANYA.AI

Context window of an LLM

is the maximum number of tokens
it can consider at once
when generating text

Token length refers to the length of the input, measured in tokens.

How much information can be included in a prompt depends on the model’s context length limit. Models’ maximum context length has increased rapidly in recent years. The first three generations of GPTs have 1K, 2K, and 4K context length respectively. This is barely long enough for a college essay and too short for most legal documents or research papers.

Context length expansion soon became a race among model providers and practitioners. Figure 5-3 shows how quickly the context length limit is expanding. Within 5 years, it grew 2,000 times from GPT-2’s 1K context length to Gemini-1.5 Pro’s 2M context length. 100K context length can fit a moderate-sized book. As a reference, this book contains approximately 120,000 words or 160,000 tokens. 2M context length can fit approximately 2,000 Wikipedia pages and a reasonably complex codebase such as PyTorch.

28 of 133

Fine-Tuning

Updating the weights of some or all layers of the pre-trained model to adapt it to the new task

Prompt Engineering

Prompt engineering is temporary learning during inference.

It lets us communicate with an LLM to steer its behavior without updating the model weights

RAG

For tasks that require latest knowledge after the model pretraining time cutoff or access to private knowledge base

LAVANYA.AI

29 of 133

Zero Shot & Few Shot Learning

LAVANYA.AI

30 of 133

Steer Behavior With Prompts

30

Q: Are whales fish?

A:

Model Output: No, whales are not fish. Whales are mammals. Here are some key differences between whales (mammals) and fish:

Breathing: Whales breathe air through lungs and must come to the surface to breathe. Fish, on the other hand, extract oxygen from water through their gills.

Body Temperature: Whales are warm-blooded, meaning they can regulate their body temperature internally. Fish are cold-blooded and their body temperature varies with the environment.

Reproduction: Whales give birth to live young and nurse them with milk. Most fish lay eggs, although there are some fish that give birth to live young.

Skin: Whales have smooth skin with a layer of blubber underneath for insulation. Fish have scales covering their bodies.

Movement: Whales move their tails up and down when swimming, while fish move their tails side to side.

These distinctions categorize whales as mammals and not fish.

31 of 133

Steer Behavior With Prompts

Q: Are whales fish?

A: No

Q: Are pig-nosed turtle mammals?

A:

Model Output: No

LAVANYA.AI

32 of 133

Steer Behavior With Prompts

Q: Are whales fish?

A: No, they're mammals.

Q: Are pig-nosed turtle mammals?

A:

Model Output: No, they're reptiles

Q: Are whales fish?

A: No

Q: Are pig-nosed turtle mammals?

A:

Model Output: No

LAVANYA.AI

33 of 133

Few-Shot Prompting

33

Classify the text delimited by triple backticks into one of the following classes.

Classes: {desired_tags}

Text: ```{ticket_text1}```

Class: question

Text: ```{ticket_text2}```

Class: type_bug

<...more examples here>

Text: ```{ticket_text}```

Class:

LAVANYA.AI

Just prompting improves model behavior a lot!

Few-shot presents a set of high-quality demonstrations, each consisting of both input and desired output, on the target task.

As the model first sees good examples, it can better understand human intention and criteria for what kinds of answers are wanted. Therefore, few-shot learning often leads to better performance than zero-shot. However, it comes at the cost of more token consumption and may hit the context length limit when input and output text are long.

One thing to note here, we are limited in how many examples we can give the model by the size of the context windows. If our model doesn’t perform well after adding 10s of examples (exact number depends on the size of the examples - e.g. are you sending just paragraphs or text or whole docs), we should try fine-tuning the model instead.

34 of 133

Each example in the prompt is a shot.

More powerful models

...need fewer examples

...to have better performance

LAVANYA.AI

Prompting with examples = better performance

Each example provided in the prompt is called a shot. Teaching a model to learn from examples in the prompt is also called few-shot learning. With five examples, it’s five-shot learning. When no example is provided, it’s zero-shot learning.

Exactly how many examples are needed depends on the model and the application. You’ll need to experiment to determine the optimal number of examples for your applications. In general, the more examples are provided, the better the model can learn. The number of examples is limited by the model’s maximum context length. The more examples there are, the longer your prompt will be, increasing the inference cost.

For GPT-3 (2020), few-shot learning showed significant improvement compared to zero-shot learning. However, for GPT-4 and a few other models in Microsoft’s 2023 analysis, few-shot learning led to only limited improvement compared to zero-shot learning. This result suggests that as models become more powerful, they become better at understanding and following instructions, which leads to better performance with fewer examples.

35 of 133

Is prompting

useful on its own?

35

2. Clean the data – sheet

3. Ask questions – ChatGPT

1. Get the data – docs

LAVANYA.AI

Let’s try it!

36 of 133

Prompt

Structure

LAVANYA.AI

37 of 133

LAVANYA.AI

System &

User Prompts

SYSTEM

The system role sets the behavior of the assistant

ASSISTANT

ChatGPT (LLM) receives the prompt and returns the completions

USER

The user (You) sends the prompts and receives the completions

System

User

Assistant

context

prompt

response

Prompt structure

system prompt

task description

sets the tone�

user prompt

task

specific question�

assistant

response

Examples

Many model APIs give you the option to split a prompt into system prompt and user prompt. You can think of the system prompt as the task description and the user prompt as the task. Let’s go through an example to see what this looks like.

Imagine you want to build a chatbot that helps buyers understand property disclosures. A user can upload a disclosure and ask questions such as “How old is the roof?” or “What is unusual about this property?”. You want this chatbot to act like a real estate agent. You can put this roleplaying instruction in the system prompt while the user question and the uploaded disclosure can be in the user prompt.

38 of 133

LAVANYA.AI

System prompt

= task description

User prompt

= the task

example from OpenAI’s docs

System prompt =

what the LLM can do
how it should behave

User prompt

the question to answer

For tasks in which lots of independent sets of instructions are needed to handle different cases, it can be beneficial to first classify the type of query and to use that classification to determine which instructions are needed. This can be achieved by defining fixed categories and hardcoding instructions that are relevant for handling tasks in a given category. This process can also be applied recursively to decompose a task into a sequence of stages. The advantage of this approach is that each query will contain only those instructions that are required to perform the next stage of a task which can result in lower error rates compared to using a single query to perform the whole task. This can also result in lower costs since larger prompts cost more to run (see pricing information).

Suppose for example that for a customer service application, queries could be usefully classified as follows:

Based on the classification of the customer query, a set of more specific instructions can be provided to a model for it to handle next steps. For example, suppose the customer requires help with "troubleshooting".

39 of 133

LAVANYA.AI

System prompt = task description

User prompt = task

Another example

System prompt =

tone
how it should behave

User prompt

the question to answer

System prompts are given more attention

comes first in the final prompt

model better at paying attention to what comes first

post-trained to pay more attention to the system prompt

Examples

Many model APIs give you the option to split a prompt into system prompt and user prompt. You can think of the system prompt as the task description and the user prompt as the task. Let’s go through an example to see what this looks like.

Imagine you want to build a chatbot that helps buyers understand property disclosures. A user can upload a disclosure and ask questions such as “How old is the roof?” or “What is unusual about this property?”. You want this chatbot to act like a real estate agent. You can put this roleplaying instruction in the system prompt while the user question and the uploaded disclosure can be in the user prompt.

40 of 133

LAVANYA.AI

Is all context equal?

All parts of a prompt are not equal.

Models are much better at understanding instructions

given at the beginning and
the end of a prompt
than in the middle.

from Shreya’s book - AI engineering

One way to evaluate the effectiveness of different parts of a prompt is to use a test commonly known as the needle in a haystack (NIAH). The idea is to insert a random piece of information (the needle) in different locations in a prompt (the haystack) and ask the model to find it. Figure 5-4 shows an example of a piece of information used in Liu et al.’s paper.

All the models tested seemed much better at finding the information when it’s closer to the beginning and the end of the prompt than in the middle.

For example, if you have the transcript of a long doctor visit, you can ask the model to return information mentioned throughout the meeting, such as the drug the patient is using or the blood type of the patient.7 Make sure that the information you use to test is private to avoid the possibility of it being included in the model’s training data. If that’s the case, a model might just rely on its internal knowledge, instead of the context, to answer the question.

Similar tests can also be used to evaluate how good a model is at processing long prompts. If the model’s performance grows increasingly worse with a longer context, then perhaps you should find a way to shorten your prompts.

41 of 133

Temperature adjusts how adventurous the model is with its word choices by reshaping the probability landscape

High ~ Chatbots: Makes responses more varied and human-like, enhancing engagement

Low ~ Code Generation: Produces more predictable and stable outputs, reducing errors.

LAVANYA.AI

Temperature

42 of 133

nat.dev

42

43 of 133

Best

Practices

LAVANYA.AI

44 of 133

LAVNYA.AI

Write clear and explicit instructions

Like a smart intern that needs instructions

Explain, without ambiguity, what you want the model to do

Include details to get more relevant answers

Communicating with AI is the same as communicating with humans: clarity helps. Here are a few tips on how to write clear instructions.

These models can’t read your mind. If outputs are too long, ask for brief replies. If outputs are too simple, ask for expert-level writing. If you dislike the format, demonstrate the format you’d like to see. The less the model has to guess at what you want, the more likely you’ll get it.

Explain, without ambiguity, what you want the model to do

If you want the model to score an essay, explain what the score system you want to use. Is it from 1 to 5 or 1 to 10? If there’s an essay the model’s uncertain about, do you want it to pick a score to the best of its ability or to output “I don’t know”?

As you experiment with a prompt, you might observe undesirable behaviors that require adjustments to the prompt to prevent them. For example, if the model outputs fractional scores (4.5) and you don’t want fractional scores, update your prompt to tell the model to output only integer scores.

Tactic: Include details in your query to get more relevant answers

In order to get a highly relevant response, make sure that requests provide any important details or context. Otherwise you are leaving it up to the model to guess what you mean.

45 of 133

LAVANYA.AI

Use delimiters to clearly indicate distinct parts of the input

46 of 133

Specify the Output Format

LAVANYA.AI

Specify the output format�Specially if you want to chain the models

If you want the model to be concise, tell it so. Not only long outputs are costly (model APIs charge per token) but also increase latency. If the model tends to begin its response with preambles such as “Based on the content of this essay, I’d give it a score of”, make explicit that you don’t want preambles.

Ensuring the model outputs are in the correct format is essential when they are used by downstream applications that require specific formats. If you want the model to generate JSON, specify what the keys in the JSON should be. Give examples if necessary.

For tasks expecting structured outputs, such as classification, use markers to mark the end of the prompts to let the model knows that the structured outputs should begin.8 Without markers, the model might continue appending to the input, as shown in Table 5-2. Make sure to choose markers that are unlikely to appear in your inputs. Otherwise, the model might get confused.

Table 5-2: Without explicit markers to mark the end of the input, a model might continue appending to it instead of generating structured outputs.

47 of 133

Ask the model to adopt a persona

LAVANYA.AI

48 of 133

Provide examples

LAVANYA.AI

Provide examples – like we saw earlier

Providing general instructions that apply to all examples is generally more efficient than demonstrating all permutations of a task by example, but in some cases providing examples may be easier. For example, if you intend for the model to copy a particular style of responding to user queries which is difficult to describe explicitly. This is known as "few-shot" prompting.�

Examples can reduce ambiguity about how you want the model to respond. Imagine you’re building a bot to talk to young children. Given the question “Will Santa bring me presents on Christmas?”, a model might respond that Santa Claus is a fictional character and therefore, can’t bring anyone Christmas presents. This answer wouldn’t endear your bot to your users.

To prevent this, you can provide the model with examples of how to respond to questions about fictional characters, such as saying that tooth fairies exist. Figure 5-7 demonstrates the power of examples in Claude’s prompt engineering tutorial.

This might sound obvious, but if you’re worried about input token length, opt for example formats that use fewer tokens.

49 of 133

Bring your own texts as context

LAVANYA.AI

The model doesn’t know about your docs by default�Add your docs to the prompt�We’ll see how to do this with RAG

Provide sufficient context

Instruct the model to answer using a reference text�If we can provide a model with trusted information that is relevant to the current query, then we can instruct the model to use the provided information to compose its answer.

Given that all models have limited context windows, we need some way to dynamically lookup information that is relevant to the question being asked. Embeddings can be used to implement efficient knowledge retrieval. See the tactic "Use embeddings-based search to implement efficient knowledge retrieval" for more details on how to implement this.

Instruct the model to answer with citations from a reference text

If the input has been supplemented with relevant knowledge, it's straightforward to request that the model add citations to its answers by referencing passages from provided documents. Note that citations in the output can then be verified programmatically by string matching within the provided documents.

Just like how reference texts can help students do better on an exam, sufficient context can help models perform better. If you want the model to answer questions about a paper, including that paper in the context will likely improve the model’s responses. Context can also mitigate hallucinations. If the model isn’t provided with the necessary information, it’ll have to rely on its internal knowledge, which might be unreliable, causing it to hallucinate.

How to restrict a model’s knowledge to only its context

In many scenarios, it’s desirable for the model to use only information provided in the context to respond. This is especially common for roleplaying and other simulations. For example, if you want a model to play a character in the game Skyrim, this character should only know about the Skyrim universe and shouldn’t be able to answer questions like: “What’s your favorite Starbucks item?”

How to restrict a model to only the context is tricky. Clear instructions, such as “answer using only the provided context,” along with examples of questions it shouldn’t be able to answer, can help. You can also instruct the model to specifically quote where in the provided corpus it draws its answer from. This approach can nudge the model to generate only answers that are supported by the context.

However, since there’s no guarantee that the model will follow all instructions, prompting alone may not reliably produce the desired outcome. Fine-tuning a model on your own corpus is another option, but pretraining data can still leak into its responses. The safest method is to train a model exclusively on the permitted corpus of knowledge, though this is often not feasible for most use cases. Additionally, the corpus may be too limited to train a high-quality model.

You can either provide the model with the necessary context or give it tools to gather context. The process of gathering necessary context for a given query is called context construction. Context construction tools include data retrieval, such as in a RAG pipeline, and web search. These tools will be discussed later in this chapter.

50 of 133

Ask the model if it missed anything on previous passes

LAVANYA.AI

51 of 133

Break complex tasks into simpler subtasks

LAVANYA.AI

Break complex tasks into simpler subtasks

Some tasks are best specified as a sequence of steps. Writing the steps out explicitly can make it easier for the model to follow them.

For complex tasks that require multiple steps, break those tasks into subtasks. Instead of having one giant prompt for the whole task, each subtask has its own prompt. These subtasks are then chained together. Consider a customer support chatbot. The process of responding to a customer request can be decomposed into two steps:

Intent classification: identify the intent of the request.

Use intent classification to identify the most relevant instructions for a user query

For tasks in which lots of independent sets of instructions are needed to handle different cases, it can be beneficial to first classify the type of query and to use that classification to determine which instructions are needed. This can be achieved by defining fixed categories and hardcoding instructions that are relevant for handling tasks in a given category. This process can also be applied recursively to decompose a task into a sequence of stages. The advantage of this approach is that each query will contain only those instructions that are required to perform the next stage of a task which can result in lower error rates compared to using a single query to perform the whole task. This can also result in lower costs since larger prompts cost more to run (see pricing information).

Suppose for example that for a customer service application, queries could be usefully classified as follows:

Image 1

Based on the classification of the customer query, a set of more specific instructions can be provided to a model for it to handle next steps. For example, suppose the customer requires help with "troubleshooting".

Image 2

Notice that the model has been instructed to emit special strings to indicate when the state of the conversation changes. This enables us to turn our system into a state machine where the state determines which instructions are injected. By keeping track of state, what instructions are relevant at that state, and also optionally what state transitions are allowed from that state, we can put guardrails around the user experience that would be hard to achieve with a less structured approach.��Generating response: based on this intent, instruct the model on how to respond. If there are ten possible intents, you’ll need ten different prompts.

The following example from OpenAI’s prompt engineering guide shows the intent classification prompt and the prompt for one intent (troubleshooting). The prompts are lightly modified for brevity.

Notice that the model has been instructed to emit special strings to indicate when the state of the conversation changes. This enables us to turn our system into a state machine where the state determines which instructions are injected. By keeping track of state, what instructions are relevant at that state, and also optionally what state transitions are allowed from that state, we can put guardrails around the user experience that would be hard to achieve with a less structured approach.

Given this example, you might wonder: why not further decompose the intent classification prompt into two prompts, one for the primary category and one for the second category. How small each subtask should be depends on each use case and the performance, cost, and latency tradeoff you’re comfortable with. You’ll need to experiment to find the optimal decomposition and chaining.

While models are getting better at understanding complex instructions, they are still better with simpler ones. Prompt decomposition not only enhances performance but also offers several additional benefits.

Monitoring: you can monitor not just the final output but also all intermediate outputs.

Debugging: you can isolate the step that is having trouble and fix it independently without changing the model’s behavior at the other steps.

Parallelization: independent steps can sometimes be executed in parallel to save time. Imagine asking a model to generate three different story versions for three different reading levels: first grade, eighth grade, and college freshman. All these three versions can be generated at the same time, significantly reducing the output latency.9

Effort: it’s easier to write simple prompts than complex prompts.

One downside of prompt decomposition is that it can increase the latency perceived by users, especially for tasks where users don’t see the intermediate outputs. With more intermediate steps, users have to wait longer to see the first output token generated in the final step.

Prompt decomposition typically involves more model queries, which can increase costs. However, the cost of two decomposed prompts might not be twice that of one original prompt. This is because most model APIs charge per input and output token, and smaller prompts often incur fewer tokens. Additionally, you can use cheaper models for simpler steps. For example, in customer support, it’s common to use a weaker model for intent classification and a stronger model to generate user responses. Even if the cost increases, the improved performance and reliability can make it worthwhile.

As you work to improve your application, your prompt can quickly become complex. You might need to provide more detailed instructions, add more examples, and consider edge cases. GoDaddy (2024) found that the prompt for their customer support chatbot bloated to over 1,500 tokens after one iteration. After decomposing the prompt into smaller prompts targeting different subtasks, they found that their model performed better while also reducing token costs.

52 of 133

Give the model time to think

❌

✅

LAVANYA.AI

53 of 133

Few-Shot

Show it reasoning examples

Chain of Thought

LAVANYA.AI

54 of 133

o1 from OpenAI

LAVANYA.AI

55 of 133

Project

LAVANYA.AI

56 of 133

Project

56

Data analysis with ChatGPT

LAVANYA.AI

57 of 133

Challenge

(10 mins)

57

2. Go to ChatGPT

Paste in data
Ask questions
Get insights

1. Get the data wandb.me/class-data

Plot a distribution of Scores
Make a correlation matrix of features
Amazon Reviews Dataset

what types of products get mostly good and bad reviews?
which products have generally good and bad reviews
which reviewers give generally good and bad reviews

Wines Dataset

is wine price correlated to score?
what types of wines get the best scores?
which regions have the best wines?
are some reviewers harsher at rating than others?

Question Ideas

Gumption

LAVANYA.AI

58 of 133

Limited Context

We can’t feed it all our data

How do we judge if it’s right?

Rigorous evaluations for all expected use cases

How did it get better?

Save prompts, models

LAVANYA.AI

Challenges

59 of 133

RAG

LAVANYA.AI

60 of 133

Thank You

Learn more at lavanya.ai

2 Prompt Engineering

61 of 133

62 of 133

GenAI Enterprise Accelerator Master Class

Prompt Engineering

63 of 133

Industry leading partnerships

Streamlined End-to-End ML & LLM Workflow

Technology

AI Developer Platform

Customers

1000+ Enterprises & AI Leaders

Leadership

Community & Ecosystem

Ecosystem >50,000 repo & framework integrations

Investors

Weights & Biases

Mission: Build the best tools for AI developers

Series C: $250m in funding

W&B Models

W&B Weave

W&B Core

64 of 133

Are you using GenAI today personally?

64

65 of 133

Are you exploring GenAI as a company?

65

66 of 133

Are you using GenAI

in production?

66

67 of 133

Prototype To Production

Nov 4

Foundations of Prompting, RAG systems, Agents

Nov 25

Evaluate and Productionize AI Systems

Dec 16

Build and deploy a GenAI app end-to-end

68 of 133

Community

68

wandb.me/discord

ACCELERATOR.CC

69 of 133

Ask Qs!

69

70 of 133

GenAI Accelerator was transformational for our team. Weights & Biases' depth of knowledge on GenAI is unmatched—it's clear why nearly every foundation model is built on their platform. Highly recommended for anyone serious about leveraging GenAI in their business.”

We will help you train your teams on GenAI best practices

accelerator.cc/class

James Cham

Partner, Bloomberg Beta

71 of 133

We will help you build production quality applications

The support we’ve received from Weights & Biases has been exceptional. They have been invaluable as thought partners and guides throughout our journey. Their deep expertise not only allowed us to anticipate potential roadblocks well in advance but also equipped us with well-considered solutions. Thanks to their guidance, we were able to bring the product to market more swiftly and with fewer challenges.”

accelerator.cc

Weave

Deliver GenAI with confidence

Monitor

Collaborate

Evaluate

Build

David Rogier

CEO, MasterClass

72 of 133

Prompt Engineering

1

Agenda

PE – Play

2

RAG

3

RAG – Play

4

Feedback

5

GENAI ACCELERATOR

73 of 133

73

AI

Machine Learning

Deep Learning

LLMs

GenAI

Foundation Models

74 of 133

ACCELERATOR.CC

What’s so exciting?

75 of 133

LLMs massively expedite the ability for anyone to quickly build useful applications.

ACCELERATOR.CC

incredibly performant out of the box
bring down the pace of iteration from months to hours

This is because they can be so incredibly performant out of the box, with just a little bit of prompt engineering. They bring down the pace of iteration from months to hours!

With traditional ML, the steps for putting a useful model in production (getting data, training the model and deploying) took months. With LLMs, you can write good prompts and iterate to find one that works for you in a matter of minutes-hours. You can then start making API calls to the model and making inferences in a couple hours! Not needing massive amounts of data to learn new concepts, and being able to tune your prompts on a small number of examples to get massive lifts in performance is a gamer changer too!

76 of 133

Getting great results from LLMs doesn’t require knowledge of deep learning models, programming, math or statistics.

ACCELERATOR.CC

77 of 133

Democratization of AI

Foundation

Model Builder

ML

Engineer

Software

Developer

1000s

100,000s

10,000,000s

ACCELERATOR.CC

78 of 133

Your IP is all the things you tried along the way

ACCELERATOR.CC

79 of 133

These models are like zip files

compressing the knowledge of the entire internet

ACCELERATOR.CC

80 of 133

ACCELERATOR.CC

Transformers 🤿

81 of 133

2017

ACCELERATOR.CC

82 of 133

Self Attention

ACCELERATOR.CC

83 of 133

Multi-Headed Attention

84 of 133

Tokenization

ACCELERATOR.CC

85 of 133

Tokenizers

wandb.me/tokenizer

85

ACCELERATOR.CC

86 of 133

How much information can be included in a prompt depends on the model’s context length.

ACCELERATOR.CC

Context is memory

Context Window

Length

Token

Context window of an LLM

is the maximum number of tokens
it can consider at once
when generating text

Token length refers to the length of the input, measured in tokens.

How much information can be included in a prompt depends on the model’s context length limit. Models’ maximum context length has increased rapidly in recent years. The first three generations of GPTs have 1K, 2K, and 4K context length respectively. This is barely long enough for a college essay and too short for most legal documents or research papers.

Context length expansion soon became a race among model providers and practitioners. Figure 5-3 shows how quickly the context length limit is expanding. Within 5 years, it grew 2,000 times from GPT-2’s 1K context length to Gemini-1.5 Pro’s 2M context length. 100K context length can fit a moderate-sized book. As a reference, this book contains approximately 120,000 words or 160,000 tokens. 2M context length can fit approximately 2,000 Wikipedia pages and a reasonably complex codebase such as PyTorch.

87 of 133

How much information can be included in a prompt depends on the model’s context length.

ACCELERATOR.CC

Context is memory

88 of 133

ACCELERATOR.CC

Fine-Tuning

Updating the weights of some or all layers of the pre-trained model to adapt it to the new task

Prompt Engineering

Prompt engineering is temporary learning during inference.

It lets us communicate with an LLM to steer its behavior without updating the model weights

RAG

For tasks that require latest knowledge after the model pretraining time cutoff or access to private knowledge base

89 of 133

ACCELERATOR.CC

Zero Shot & Few Shot

Learning

90 of 133

Steer Behavior With Prompts

90

Q: Are whales fish?

A:

Model Output: No, whales are not fish. Whales are mammals. Here are some key differences between whales (mammals) and fish:

Breathing: Whales breathe air through lungs and must come to the surface to breathe. Fish, on the other hand, extract oxygen from water through their gills.

Body Temperature: Whales are warm-blooded, meaning they can regulate their body temperature internally. Fish are cold-blooded and their body temperature varies with the environment.

Reproduction: Whales give birth to live young and nurse them with milk. Most fish lay eggs, although there are some fish that give birth to live young.

Skin: Whales have smooth skin with a layer of blubber underneath for insulation. Fish have scales covering their bodies.

Movement: Whales move their tails up and down when swimming, while fish move their tails side to side.

These distinctions categorize whales as mammals and not fish.

91 of 133

Steer Behavior With Prompts

Q: Are whales fish?

A: No

Q: Are pig-nosed turtle mammals?

A:

Model Output: No

ACCELERATOR.CC

92 of 133

Steer Behavior With Prompts

Q: Are whales fish?

A: No, they're mammals.

Q: Are pig-nosed turtle mammals?

A:

Model Output: No, they're reptiles

Q: Are whales fish?

A: No

Q: Are pig-nosed turtle mammals?

A:

Model Output: No

ACCELERATOR.CC

93 of 133

Few-Shot Prompting

93

Classify the text delimited by triple backticks into one of the following classes.

Classes: {desired_tags}

Text: ```{ticket_text1}```

Class: question

Text: ```{ticket_text2}```

Class: type_bug

<...more examples here>

Text: ```{ticket_text}```

Class:

ACCELERATOR.CC

Just prompting improves model behavior a lot!

Few-shot presents a set of high-quality demonstrations, each consisting of both input and desired output, on the target task.

As the model first sees good examples, it can better understand human intention and criteria for what kinds of answers are wanted. Therefore, few-shot learning often leads to better performance than zero-shot. However, it comes at the cost of more token consumption and may hit the context length limit when input and output text are long.

One thing to note here, we are limited in how many examples we can give the model by the size of the context windows. If our model doesn’t perform well after adding 10s of examples (exact number depends on the size of the examples - e.g. are you sending just paragraphs or text or whole docs), we should try fine-tuning the model instead.

94 of 133

Each example in the prompt is a shot.

More powerful models

...need fewer examples

...to have better performance

ACCELERATOR.CC

Prompting with examples = better performance

Each example provided in the prompt is called a shot. Teaching a model to learn from examples in the prompt is also called few-shot learning. With five examples, it’s five-shot learning. When no example is provided, it’s zero-shot learning.

Exactly how many examples are needed depends on the model and the application. You’ll need to experiment to determine the optimal number of examples for your applications. In general, the more examples are provided, the better the model can learn. The number of examples is limited by the model’s maximum context length. The more examples there are, the longer your prompt will be, increasing the inference cost.

For GPT-3 (2020), few-shot learning showed significant improvement compared to zero-shot learning. However, for GPT-4 and a few other models in Microsoft’s 2023 analysis, few-shot learning led to only limited improvement compared to zero-shot learning. This result suggests that as models become more powerful, they become better at understanding and following instructions, which leads to better performance with fewer examples.

95 of 133

Is prompting

useful on its own?

95

2. Clean the data – sheet

3. Ask questions – ChatGPT

1. Get the data – docs

ACCELERATOR.CC

Let’s try it!

96 of 133

ACCELERATOR.CC

Prompt Structure

97 of 133

ACCELERATOR.CC

System &

User Prompts

System

The system role sets the behavior of the assistant

Assistant

ChatGPT (LLM) receives the prompt and returns the completions

User

The user (You) sends the prompts and receives the completions

System

User

Assistant

context

prompt

response

Prompt structure

system prompt

task description

sets the tone�

user prompt

task

specific question�

assistant

response

Examples

Many model APIs give you the option to split a prompt into system prompt and user prompt. You can think of the system prompt as the task description and the user prompt as the task. Let’s go through an example to see what this looks like.

Imagine you want to build a chatbot that helps buyers understand property disclosures. A user can upload a disclosure and ask questions such as “How old is the roof?” or “What is unusual about this property?”. You want this chatbot to act like a real estate agent. You can put this roleplaying instruction in the system prompt while the user question and the uploaded disclosure can be in the user prompt.

98 of 133

ACCELERATOR.CC

System prompt

= task description

User prompt

= the task

example from OpenAI’s docs

System prompt =

what the LLM can do
how it should behave

User prompt

the question to answer

For tasks in which lots of independent sets of instructions are needed to handle different cases, it can be beneficial to first classify the type of query and to use that classification to determine which instructions are needed. This can be achieved by defining fixed categories and hardcoding instructions that are relevant for handling tasks in a given category. This process can also be applied recursively to decompose a task into a sequence of stages. The advantage of this approach is that each query will contain only those instructions that are required to perform the next stage of a task which can result in lower error rates compared to using a single query to perform the whole task. This can also result in lower costs since larger prompts cost more to run (see pricing information).

Suppose for example that for a customer service application, queries could be usefully classified as follows:

Based on the classification of the customer query, a set of more specific instructions can be provided to a model for it to handle next steps. For example, suppose the customer requires help with "troubleshooting".

99 of 133

ACCELERATOR.CC

System prompt = task description

User prompt = task

Another example

System prompt =

tone
how it should behave

User prompt

the question to answer

System prompts are given more attention

comes first in the final prompt

model better at paying attention to what comes first

post-trained to pay more attention to the system prompt

Examples

Many model APIs give you the option to split a prompt into system prompt and user prompt. You can think of the system prompt as the task description and the user prompt as the task. Let’s go through an example to see what this looks like.

Imagine you want to build a chatbot that helps buyers understand property disclosures. A user can upload a disclosure and ask questions such as “How old is the roof?” or “What is unusual about this property?”. You want this chatbot to act like a real estate agent. You can put this roleplaying instruction in the system prompt while the user question and the uploaded disclosure can be in the user prompt.

100 of 133

ACCELERATOR.CC

Is all context equal?

All parts of a prompt are not equal.

Models are much better at understanding instructions

given at the beginning and
the end of a prompt
than in the middle.

from Shreya’s book - AI engineering

One way to evaluate the effectiveness of different parts of a prompt is to use a test commonly known as the needle in a haystack (NIAH). The idea is to insert a random piece of information (the needle) in different locations in a prompt (the haystack) and ask the model to find it. Figure 5-4 shows an example of a piece of information used in Liu et al.’s paper.

All the models tested seemed much better at finding the information when it’s closer to the beginning and the end of the prompt than in the middle.

For example, if you have the transcript of a long doctor visit, you can ask the model to return information mentioned throughout the meeting, such as the drug the patient is using or the blood type of the patient.7 Make sure that the information you use to test is private to avoid the possibility of it being included in the model’s training data. If that’s the case, a model might just rely on its internal knowledge, instead of the context, to answer the question.

Similar tests can also be used to evaluate how good a model is at processing long prompts. If the model’s performance grows increasingly worse with a longer context, then perhaps you should find a way to shorten your prompts.

101 of 133

Temperature adjusts how adventurous the model is with its word choices by reshaping the probability landscape

High ~ Chatbots: Makes responses more varied and human-like, enhancing engagement

Low ~ Code Generation: Produces more predictable and stable outputs, reducing errors.

ACCELERATOR.CC

Temperature

102 of 133

nat.dev

102

103 of 133

ACCELERATOR.CC

Best Practices

104 of 133

ACCELERATOR.CC

Write clear and explicit instructions

Like a smart intern that needs instructions

Explain, without ambiguity, what you want the model to do

Include details to get more relevant answers

Communicating with AI is the same as communicating with humans: clarity helps. Here are a few tips on how to write clear instructions.

These models can’t read your mind. If outputs are too long, ask for brief replies. If outputs are too simple, ask for expert-level writing. If you dislike the format, demonstrate the format you’d like to see. The less the model has to guess at what you want, the more likely you’ll get it.

Explain, without ambiguity, what you want the model to do

If you want the model to score an essay, explain what the score system you want to use. Is it from 1 to 5 or 1 to 10? If there’s an essay the model’s uncertain about, do you want it to pick a score to the best of its ability or to output “I don’t know”?

As you experiment with a prompt, you might observe undesirable behaviors that require adjustments to the prompt to prevent them. For example, if the model outputs fractional scores (4.5) and you don’t want fractional scores, update your prompt to tell the model to output only integer scores.

Tactic: Include details in your query to get more relevant answers

In order to get a highly relevant response, make sure that requests provide any important details or context. Otherwise you are leaving it up to the model to guess what you mean.

105 of 133

ACCELERATOR.CC

Use delimiters to clearly indicate distinct parts of the input

106 of 133

ACCELERATOR.CC

Specify the Output Format

Specify the output format�Specially if you want to chain the models

If you want the model to be concise, tell it so. Not only long outputs are costly (model APIs charge per token) but also increase latency. If the model tends to begin its response with preambles such as “Based on the content of this essay, I’d give it a score of”, make explicit that you don’t want preambles.

Ensuring the model outputs are in the correct format is essential when they are used by downstream applications that require specific formats. If you want the model to generate JSON, specify what the keys in the JSON should be. Give examples if necessary.

For tasks expecting structured outputs, such as classification, use markers to mark the end of the prompts to let the model knows that the structured outputs should begin.8 Without markers, the model might continue appending to the input, as shown in Table 5-2. Make sure to choose markers that are unlikely to appear in your inputs. Otherwise, the model might get confused.

Table 5-2: Without explicit markers to mark the end of the input, a model might continue appending to it instead of generating structured outputs.

107 of 133

ACCELERATOR.CC

Ask the model to adopt a persona

108 of 133

ACCELERATOR.CC

Provide examples

Provide examples – like we saw earlier

Providing general instructions that apply to all examples is generally more efficient than demonstrating all permutations of a task by example, but in some cases providing examples may be easier. For example, if you intend for the model to copy a particular style of responding to user queries which is difficult to describe explicitly. This is known as "few-shot" prompting.�

Examples can reduce ambiguity about how you want the model to respond. Imagine you’re building a bot to talk to young children. Given the question “Will Santa bring me presents on Christmas?”, a model might respond that Santa Claus is a fictional character and therefore, can’t bring anyone Christmas presents. This answer wouldn’t endear your bot to your users.

To prevent this, you can provide the model with examples of how to respond to questions about fictional characters, such as saying that tooth fairies exist. Figure 5-7 demonstrates the power of examples in Claude’s prompt engineering tutorial.

This might sound obvious, but if you’re worried about input token length, opt for example formats that use fewer tokens.

109 of 133

ACCELERATOR.CC

Bring your own texts as context

The model doesn’t know about your docs by default�Add your docs to the prompt�We’ll see how to do this with RAG

Provide sufficient context

Instruct the model to answer using a reference text�If we can provide a model with trusted information that is relevant to the current query, then we can instruct the model to use the provided information to compose its answer.

Given that all models have limited context windows, we need some way to dynamically lookup information that is relevant to the question being asked. Embeddings can be used to implement efficient knowledge retrieval. See the tactic "Use embeddings-based search to implement efficient knowledge retrieval" for more details on how to implement this.

Instruct the model to answer with citations from a reference text

If the input has been supplemented with relevant knowledge, it's straightforward to request that the model add citations to its answers by referencing passages from provided documents. Note that citations in the output can then be verified programmatically by string matching within the provided documents.

Just like how reference texts can help students do better on an exam, sufficient context can help models perform better. If you want the model to answer questions about a paper, including that paper in the context will likely improve the model’s responses. Context can also mitigate hallucinations. If the model isn’t provided with the necessary information, it’ll have to rely on its internal knowledge, which might be unreliable, causing it to hallucinate.

How to restrict a model’s knowledge to only its context

In many scenarios, it’s desirable for the model to use only information provided in the context to respond. This is especially common for roleplaying and other simulations. For example, if you want a model to play a character in the game Skyrim, this character should only know about the Skyrim universe and shouldn’t be able to answer questions like: “What’s your favorite Starbucks item?”

How to restrict a model to only the context is tricky. Clear instructions, such as “answer using only the provided context,” along with examples of questions it shouldn’t be able to answer, can help. You can also instruct the model to specifically quote where in the provided corpus it draws its answer from. This approach can nudge the model to generate only answers that are supported by the context.

However, since there’s no guarantee that the model will follow all instructions, prompting alone may not reliably produce the desired outcome. Fine-tuning a model on your own corpus is another option, but pretraining data can still leak into its responses. The safest method is to train a model exclusively on the permitted corpus of knowledge, though this is often not feasible for most use cases. Additionally, the corpus may be too limited to train a high-quality model.

You can either provide the model with the necessary context or give it tools to gather context. The process of gathering necessary context for a given query is called context construction. Context construction tools include data retrieval, such as in a RAG pipeline, and web search. These tools will be discussed later in this chapter.

110 of 133

ACCELERATOR.CC

Ask the model if it missed anything on previous passes

111 of 133