Local Explanations for �Deep Learning Models
Uncertainty (Part 2); Prompting; Chain-of-Thoughts
Ana Marasović
University of Utah
Reminders
1st HW due: Tomorrow 11:59p
You’re allowed to be late twice* up to 48H without a penalty, no need to notify me
* Does not apply to in-person activities
�Drop deadline: This Friday, Sept 1
�Next Monday (Sept 4): holiday, so no class
�1st paper discussion on Sept 11 (Monday): https://utah-explainability.github.io/assignments/paper_discussions/
UGs - let me know whether you want to participate in all roles or only discussion assistant ones
What did we talk about last Monday
Simple post-hoc calibration methods
Temperature scaling
For softmax-based classification tasks, we can post-hoc rescale the logits
[Guo et al., 17] initialize a single temperature parameter T
More generally, temperature scaling is a simple extension of Platt scaling [Platt, 99]
Slide source: COLING’22 tutorial
Temperature scaling
For softmax-based classification tasks, we can post-hoc rescale the logits
[Guo et al., 17] initialize a single temperature parameter T
How to find the right temperature? �Optimize T on a held-out calibration set to minimize the negative log-likelihood
Slide source: COLING’22 tutorial
Bayesian Approaches
Bayesian approaches
We don't just consider a single model as the definitive answer, but a whole spectrum of possible models
We assign probabilities to these models based on their compatibility with the data and our prior beliefs
The calibrated prediction is then a kind of weighted average over all these models, with more probable models having more influence
How to form a posterior distribution over neural networks?
�
Ways of expressing uncertainty
��
Slide source: COLING’22 tutorial
Conformal prediction
A motivating example: Information retrieval for fact-checking
Slide source: COLING’22 tutorial
A motivating example: Information retrieval for fact-checking
Slide source: COLING’22 tutorial
A motivating example: Information retrieval for fact-checking
Slide source: COLING’22 tutorial
A motivating example: Information retrieval for fact-checking
Slide source: COLING’22 tutorial
How conformal prediction works
Informally, conformal prediction uses “nonconformity” scores to measure surprise
Basic idea: suppose I assign a candidate label to a given input. How “strange” that this output-input pair look relative to other examples that I know to be correct?
If it is relatively strange, it is consider to be nonconforming to the dataset
It if is relatively “not that strange”, then it conforms (and we can’t rule the predicted label out)
Slide source: COLING’22 tutorial
Why is this input assigned this answer?
17
Week 2-5:
18
Why is this input assigned this answer?
In plain English, why is this input assigned this label?
Free-text explanations
Chain-of-Thoughts
Week 2-3:
Self-explaining with free-text explanations: �Given in plain language, immediately provide the gist of why is the input labeled as it is
19
Misleading because not every American over 65 can get these cards since they are not provided by Medicare, the federal health insurance program for senior citizens. They are offered as a benefit to some customers by private insurance companies that sell Medicare Advantage plans. The cards are available in limited geographic areas. Only the chronically ill qualify to use the cards for items such as food and produce.
+ documents from� the Web
💡
20
Misleading because not every American over 65 can get these cards since they are not provided by Medicare, the federal health insurance program for senior citizens. They are offered as a benefit to some customers by private insurance companies that sell Medicare Advantage plans. The cards are available in limited geographic areas. Only the chronically ill qualify to use the cards for items such as food and produce.
+ documents from� the Web
💡
21
Few-Shot Learning; �Prompting
Pretrain-then-Finetune Paradigm
23
pretrain model
finetune model
text + labels
text
Another trend: �Decrease the finetuning data size
The simplest way to do few-shot learning
24
pretrain model
finetune model
text + labels
text
Contains only few (8–16) labeled examples
A typical way to process data
The standard input-output formatting is suboptimal
A pretrained LM is well-positioned to solve the end-task if…
…we format finetuning end-task examples as similar as possible to the format used in pretraining
We add something to induce a task-specific behavior, e.g.:
We sometimes also add a task definition/instruction, e.g.,:
Passage: Trams have operated continuously in Melbourne since 1885 (the horse tram line in Fairfield opened in 1884, but was at best an irregular service). Since then they have become a distinctive part of Melbourne's character and feature in tourism and travel advertising. Melbourne's cable tram system opened in 1885, and expanded to one of the largest in the world, with of double track. The first electric tram line opened in 1889, but closed only a few years later in 1896. In 1906 electric tram systems were opened in St Kilda and Essendon, marking the start of continuous operation of Melbourne's electric trams.\n
Question: If I wanted to take a horse tram in 1884, could I look up the next tram on a schedule?\n
Answer:
Task description
A task instance
In this task, you’re expected to write answers to questions involving reasoning about negation. The answer to the question should be “yes”, “no”, “don’t know” or a phrase in the passage. Questions can have only one correct answer.\n
An example of a prompt�– A reading comprehension example
[The model generates the answer: “No”]
Example from [Ravichander et al., 2022]
Prompt-based finetuning
28
pretrain model
finetune model
text + labels
text
Contains only few (8–16) labeled examples
These examples are carefully formatted
An alternative to prompt-based finetuning
Imagine a situation where a layperson or an expert in another domain that interacts with an NLP model:
What if instead of providing few examples individually we concatenate them into one long sequence and do not change the model weights?
Passage: During the 1930s, Jehovah's Witnesses in Germany were sent to concentration camps by the thousands, due to their refusal to salute the Nazi flag, which the government considered to be a crime. Jehovah's Witnesses believe that the obligation imposed by the law of God is superior to that of laws enacted by government. Their religious beliefs include a literal version of Exodus, Chapter 20, verses 4 and 5, which says: "Thou shalt not make unto thee any graven image, or any likeness of anything that is in heaven above, or that is in the earth beneath, or that is in the water under the earth; thou shalt not bow down thyself to them nor serve them." They consider that the flag is an 'image' within this command. For this reason, they refused to salute the flag.\n
Question: Is it likely that most of these Jehovah's Witnesses survived the war (having the same likelihood of survival as other German civilians) only to later see Soviet flags in their country, or American soldiers proudly saluting the stars and stripes?\n
Answer: NO\n
###\n
Passage: Francesco Rognoni was another composer who specified the trombone in a set of divisions (variations) on the well-known song "Suzanne ung jour" (London Pro Musica, REP15). Rognoni was a master violin and gamba player whose treatise "Selva di Varie passaggi secondo l'uso moderno" (Milan 1620 and facsimile reprint by Arnaldo Forni Editore 2001) details improvisation of diminutions and Suzanne is given as one example. Although most diminutions are written for organ, string instruments or cornett, Suzanne is "per violone over Trombone alla bastarda". With virtuosic semiquaver passages across the range of the instrument, it reflects Praetorius' comments about the large range of the tenor and bass trombones, and good players of the Quartposaune (bass trombone in F) could play fast runs and leaps like a viola bastarda or cornetto. The term "bastarda" describes a technique that made variations on all the different voices of a part song, rather than just the melody or the bass: "considered legitimate because it was not polyphonic".
Question: Would you likely find the term "bastarda" regularly used in an academic paper on musical theory?\n
Answer: DON'T KNOW\n
###\n
[...]
###\n
Passage: Trams have operated continuously in Melbourne since 1885 (the horse tram line in Fairfield opened in 1884, but was at best an irregular service). Since then they have become a distinctive part of Melbourne's character and feature in tourism and travel advertising. Melbourne's cable tram system opened in 1885, and expanded to one of the largest in the world, with of double track. The first electric tram line opened in 1889, but closed only a few years later in 1896. In 1906 electric tram systems were opened in St Kilda and Essendon, marking the start of continuous operation of Melbourne's electric trams.\n
Question: If I wanted to take a horse tram in 1884, could I look up the next tram on a schedule?\n
Answer:
Task description
Examples / Shots / Demonstrations
Test instance
In this task, you’re expected to write answers to questions involving reasoning about negation. The answer to the question should be “yes”, “no”, “don’t know” or a phrase in the passage. Questions can have only one correct answer.\n
Example from [Ravichander et al., 2022]
In-context learning
This approach:
is called in-context learning
“Advanced” Prompting
Passage: Francesco Rognoni was another composer who specified the trombone in a set of divisions (variations) on the well-known song "Suzanne ung jour" (London Pro Musica, REP15). Rognoni was a master violin and gamba player whose treatise "Selva di Varie passaggi secondo l'uso moderno" (Milan 1620 and facsimile reprint by Arnaldo Forni Editore 2001) details improvisation of diminutions and Suzanne is given as one example. Although most diminutions are written for organ, string instruments or cornett, Suzanne is "per violone over Trombone alla bastarda". With virtuosic semiquaver passages across the range of the instrument, it reflects Praetorius' comments about the large range of the tenor and bass trombones, and good players of the Quartposaune (bass trombone in F) could play fast runs and leaps like a viola bastarda or cornetto. The term "bastarda" describes a technique that made variations on all the different voices of a part song, rather than just the melody or the bass: "considered legitimate because it was not polyphonic".
Question: Would you likely find the term "bastarda" regularly used in an academic paper on musical theory?
Answer: Let's think step by step. From the passage it is unclear whether 'bastarda' was a technique that was impactful and important which are reasons why one could expect to see it regularly in an academic paper on musical theory. So the answer is DON'T KNOW.
###
Passage: During the 1930s, Jehovah's Witnesses in Germany were sent to concentration camps by the thousands, due to their refusal to salute the Nazi flag, which the government considered to be a crime. Jehovah's Witnesses believe that the obligation imposed by the law of God is superior to that of laws enacted by government. Their religious beliefs include a literal version of Exodus, Chapter 20, verses 4 and 5, which says: "Thou shalt not make unto thee any graven image, or any likeness of anything that is in heaven above, or that is in the earth beneath, or that is in the water under the earth; thou shalt not bow down thyself to them nor serve them." They consider that the flag is an 'image' within this command. For this reason, they refused to salute the flag.
Question: Is it likely that most of these Jehovah's Witnesses survived the war (having the same likelihood of survival as other German civilians) only to later see Soviet flags in their country, or American soldiers proudly saluting the stars and stripes?
Answer: Let's think step by step. Worshiping any flag is forbidden by their religion and this religious law to them is superior to laws enacted by the government. Thus, even after the war, they are unlikely to condone people saluting Soviet or American flags. So the answer is NO.
###
[...]
###
Passage: Trams have operated continuously in Melbourne since 1885 (the horse tram line in Fairfield opened in 1884, but was at best an irregular service). Since then they have become a distinctive part of Melbourne's character and feature in tourism and travel advertising. Melbourne's cable tram system opened in 1885, and expanded to one of the largest in the world, with of double track. The first electric tram line opened in 1889, but closed only a few years later in 1896. In 1906 electric tram systems were opened in St Kilda and Essendon, marking the start of continuous operation of Melbourne's electric trams.
Question: If I wanted to take a horse tram in 1884, could I look up the next tram on a schedule?
Answer: Let's think step by step.
Task description
Examples / Shots with CoT
Test instance
In this task, you’re expected to write answers to questions involving reasoning about negation. The answer to the question should be “yes”, “no”, “don’t know” or a phrase in the passage. Questions can have only one correct answer.
Chain-of-thought prompting
This approach:
is called chain-of-thought prompting
Self-Consistency
FLAN-T5
Finetune a model, here T5 (an open-sourced model):
For 1.8K tasks
recent advances in prompting
Instruction finetuning data
LLaMA-Chat (and every other LLM today)
RLHF
Reliable Few-Shot Evaluation
Challenges of reliable few-shot evaluation
Sensitivity to choice of examples
Sensitivity to choice of examples
Sensitivity to choice of examples (cont.)
Estimate models’ bias toward certain answers by feeding in a “content-free” input, e.g.,:
Ideally: this input would be labelled as 50% positive and 50% negative
In practice: it’s scored 61.8% positive
“The error is contextual: a different choice of the training examples, permutation, and format will lead to different predictions for the content-free input”
Correct this error by tweaking the output matrix so that the class scores for the content-free input are uniform
No sensitivity to shuffling labels
No sensitivity to shuffling labels (cont.)
There is an ongoing discussion about these results:
Do Prompt-Based Models Really Understand the Meaning of their Prompts?
Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations (rebuttal)��How does in-context learning work? A framework for understanding the differences from traditional supervised learning (connecting a theoretical explanation of in-context learning with Min et al.’s results)
Robustness of Demonstration-based Learning Under Limited Data Scenario (more intriguing results)
Poor experimental practices
For best-practices see:
Much more on prompting can be find here:
Zero-Shot Learning
(one slide)
You still do prompting but you can not include any labeled examples, e.g.,
Obviously, there is no further training (changing of model weights)