1 of 44

Lora Aroyo, VU University, NL

Chris Welty, Google Research, NY

Quantum Collective

2 of 44

What is the probability of rolling a 2?

1/36

3 of 44

What is the probability of rolling a 3?

2/36

4 of 44

What is the probability of rolling a 4?

3/36

5 of 44

What is the probability of rolling x: 2 ≦ x ≦ 12?

1.0

6 of 44

What is the probability of rolling x: 2 ≦ x ≦ 12?

1.0

7 of 44

Count people’s answers

76% Gore OK?

58% Clinton OK?

P(likes(Clinton)) = 0.58

P(likes(Gore)) = 0.76

How do we like our leaders?

Gallup Poll, September 6–7, 1997

8 of 44

What does alabaster mean?

Count coocurrence of words in sentences on the web

find all x: P(x | w) = P(alabaster | w)

for every word w

white, pale, bone, beauty, skin

Well, the same kind of analysis is used in AI and NLP, and any kind of AI that you probably reading about today. Just to give a highly simplified example, if we want to figure out what a word means. I can simply count the coocurence of this word with all the other words on the web; and I can find a set (equivalence class) of other words that coocur with the same words that Alabaster coocurs with.

And that gives me what’s called distributional semantics, that gives me distributional understanding of what Alababaster means, it is very similar to ‘white, pale, …” as these words appear with the same words that Alabaster occurs with.

So, note here that I am not saying that alabaster occurs with White, because it doesn’t - Alabaster occurs with the same other words that WHITE occurs with

So, in order to get to the conclusion that ALABASTER means WHITE, I count, compute the marginals, and calculate the conditional probability. It is really easy. And it is basicaly the same math as polls use.

===================

And the same idea is used in AI and NLP, where we just look at the sentences on the Web and count the cooccurrences of words and from that we derive some mechanical notion of meaning.

if i want to know what the meaning of a particular work, like ALABASTER is, I count the occurrence of ALL words in English on the web, and I find other words whose probability of cooccurrence is very similar of that of alabaster. So, I find the probability of Alabaster cooccurring with every other word on the Web and what is the other set of words that have the same conditional probabilities. And I find - that the probability of WHITE cooccuring with any other word on the Web is roughly the same as ALABASTER - they are used in the same context. So I am going to derive automatically a notion of meaning, by just counting

9 of 44

all the information machines have

is all the information there is

10 of 44

there is always something else …

11 of 44

observations in psychology ...

12 of 44

Order II

Clinton OK? 53%

Gore OK? 66%

Order I

76% Gore OK?

58% Clinton OK?

Gallup Poll, September 6–7, 1997

“irrational human behavior”: order matters

So, let me give you another example that sort of proofs that there is something else going on

So, in this Gallup Poll I already mentioned, a group of psychologiest asked Gallup to experiement with something

They changed the order of the questions they asked

Now, in the way they have gathered the statistics, the order should have NOT mattered, because they had a P value small enough to assure them that they have collected enough votes that this was the probability of people liking Gore and Clinton.

But if they switched the order of the questions - so half of the people in the poll were asked whether they Like Gore first and then Clinton, and the other half of the people were asked whether they like Clinton first and then Gore.

======================

In fact, in this particular poll, which I gave as an example

Gallup ran an experiment at a request of a psychologist in which they changed the order of the two questions people we asked in the poll. And if you ask about Gore first 76% of the people like him, but if you ask about Clinton first only 66% of the people like him.

13 of 44

Order II

Clinton OK? 53%

Gore OK? 66%

Order I

76% Gore OK?

58% Clinton OK?

Gallup Poll, September 6–7, 1997

“irrational human behavior”: order matters

SENSE

14 of 44

KEEP

CALM

AND

LIKE

CLINTON

&

KEEP

CALM

AND

DON’T LIKE

CLINTON

people can’t believe contradictions

15 of 44

Statistics is not appropriate

16 of 44

trade

foreign policy

scandals

environment

economy

17 of 44

trade

foreign policy

scandals

environment

economy

There’s always something else going on

18 of 44

trade

foreign policy

scandals

environment

economy

19 of 44

This ain’t no dice game

20 of 44

Superposition of states?

21 of 44

What does alabaster mean?

count co-occurrence of words in sentences on the web

find all x: P(x | w) = P(alabaster | w)

for every word w

Eldorin, Leonorin

in Fiction

And the same thing is happening in AI

If I for example do the same as in the experiment I just described in order to figure out what the word ALABASTER means - and if I change the corpus in which I do my counting, and in stead of the web I use fictional books. And now I find that there seems to be a fictional character called ALABASTER that happens to be depicted prominently in this literature - and i get associations with other strange fantasy names = because these names occur in the same context as Alabaster, and suddenly the meaning of the word changes

And the same idea is used in AI and NLP, where we just look at the sentences on the Web and count the cooccurrences of words and from that we derive some mechanical notion of meaning.

if i want to know what the meaning of a particular work, like ALABASTER is, I count the occurrence of ALL words in English on the web, and I find other words whose probability of cooccurrence is very similar of that of alabaster. So, I find the probability of Alabaster cooccurring with every other word on the Web and what is the other set of words that have the same conditional probabilities. And I find - that the probability of WHITE cooccuring with any other word on the Web is roughly the same as ALABASTER - they are used in the same context. So I am going to derive automatically a notion of meaning, by just counting

22 of 44

What does alabaster mean?

count co-occurrence of words in sentences on the web

find all x: P(x | w) = P(alabaster | w)

for every word w

Chalk, Magnesium

in Science

23 of 44

What does alabaster mean?

count co-occurrence of words in sentences on the web

find all x: P(x | w) = P(alabaster | w)

for every word w

Chalk, Magnesium

24 of 44

What does alabaster mean?

count co-occurrence of words in sentences on the web

find all x: P(x | w) = P(alabaster | w)

for every word w

Chalk, Magnesium

25 of 44

there is always something else …

26 of 44

Statistics is is not appropriate

27 of 44

P(E) + P(¬E) = 1

P(like Clinton) = .58

P(not like Clinton) =

1 - .58 = .42

28 of 44

environment

foreign

policy

Gore

Bush

my feet hurt

Monica Lewinsky

29 of 44

Medical CrowdTruth

Chloroquine treat Malaria?

P( ) = .75

Chloroquine doesn’t treat Malaria?

P( ) = .50

We did a bunch of experiments where we asked medical experts a lot of questions

I am simplifying this dramatically, in interest of time here

And we are asking the question in the positive and in the negative

And surprisingly what we saw was that the probability of someone answering the question in the positive + the probability of someone answering the negative question did not sum to one - often it didn’t even come close

And this doesn’t make any sense

If you ask a lot of people - would you use Antibiotics to treat Typhus and Would you not use antibiotics to treat Typhus → these should sum to one as they sound like contradictory events - but they are not - because there is a lot else going on - patient’s medical history should be considered. When you ask the questions ‘would you not use Antibiotics to treat T?” it actually encourages people to think of about the reasons why they woudln’t ⇒ and this exposes a simple yes/no question to information that’s not being measured

30 of 44

P( ) = .75

P( ) = .50

.75 + .50 ≠ 1

SENSE

Probability Theory:

Chloroquine treat Malaria?

Chloroquine doesn’t treat Malaria?

31 of 44

P( ) = .75

P( ) = .50

.75 + .50 ≠ 1

SENSE

Probability Theory:

Superposition of states!

Context effect: .25

Chloroquine treat Malaria?

Chloroquine doesn’t treat Malaria?

32 of 44

ENCOURAGING DISAGREEMENT

Treats: Chloroquine, Malaria

32

For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported.

Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades.

Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood.

http://lora-aroyo.org @laroyo

33 of 44

CROWD TASK: POSITIVE

For this sentence:

Select all the RELATIONS that apply between the two HIGHLIGHTED TERMS

Is --- related to --- ?

CHLOROQUINE

MALARIA

http://lora-aroyo.org @laroyo

34 of 44

WORKER VECTOR FOR A SENTENCE

treats

associated _with

other

symptom

Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood.

http://lora-aroyo.org @laroyo

35 of 44

MANY WORKERS FOR THE SAME SENTENCE

treats

other

associated _with

symptom

Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood.

http://lora-aroyo.org @laroyo

36 of 44

ALL WORKER VECTORS

AGGREGATED IN A SENTENCE VECTOR

treats

other

none

associated _with

symptom

manifestation

side

effect

Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood.

http://lora-aroyo.org @laroyo

37 of 44

ENCOURAGING DISAGREEMENT

Treats: Chloroquine, Malaria

37

For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported.

Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades.

Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood.

http://lora-aroyo.org @laroyo

38 of 44

SENTENCE VECTORS FOR THE 3 SENTENCES

treats

other

none

associated _with

symptom

manifestation

side

effect

treats

other

none

associated _with

contraindicates

manifestation

treats

other

http://lora-aroyo.org @laroyo

39 of 44

CROWD TASK: NEGATIVE

For this sentence:

Select all the RELATIONS that DO NOT apply between the two HIGHLIGHTED TERMS

Is --- related to --- ?

CHLOROQUINE

MALARIA

http://lora-aroyo.org @laroyo

40 of 44

WORKER VECTOR FOR A SENTENCE

treats

associated _with

other

causes

9 9 9 4 9 9 8 9 3 9 9 8

0 0 0 5 0 3 1 0 1 0 0 1

Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood.

http://lora-aroyo.org @laroyo

41 of 44

WORKER VECTOR FOR A SENTENCE

treats

associated _with

other

causes

At some high volume centers, the threshold for elective repair of an ASCENDING AORTIC ANEURYSM in a patient with MARFAN SYNDROME may be even lower.

9 9 9 4 9 9 8 9 3 9 9 8

0 0 0 5 0 3 1 0 1 0 0 1

http://lora-aroyo.org @laroyo

42 of 44

P(A) + P(¬A) ≠ 1

http://lora-aroyo.org @laroyo

43 of 44

Discussion

Context Effect?

MACROSCOPIC HEMATURIA usually resolves spontaneously within days, and MICROSCOPIC HEMATURIA often occurs between mucosal infections.

1] Manhart LE, Koutsky LA. Do condoms prevent GENITAL HPV INFECTION EXTERNAL GENITAL WARTS or cervical neoplasia.

Success rates were similar between treatment groups, but there were fewer documented breakthrough FUNGAL INFECTIONS in the LIPOSOMAL AMPHOTERICIN B GROUP (215.

http://lora-aroyo.org @laroyo

44 of 44

QuantumTruth�makes sense