Shannon game results -- CS 585, 9/15/16

brenocon.com/2016ex1

http://people.cs.umass.edu/~brenocon/inlp2016/ex1_lm.pdf

Place your score in the column corresponding to your worksheet version.

score = log10 prob of revealed word

NOTE: this isn't really perplexity since typically we take 1 model and evaluate on N tokens.

instead we are taking ~30 models (number of people per version) and evaluating on the same data point over and over.

we're just doing this to illustrate how the arithmetic works.

(two sentences of context)

DATA CHECK: positive logprobs are NOT POSSIBLE. We had to delete a bunch people put in.

exp(neg avg logprob) (a.k.a. PERPLEXITY)

is the worst theoretical ppl

(if you predicted 1/6 for each word, for V=6)

NOTE: in class we did log base 10, but in the spreadsheet during class I typed exp() to calc PPL.

but it should have been 10^x since the log was base 10.

ppl = exp(-log_e prob) = 2^(-log_2 prob) = 10^(-log_10 prob)

I corrected it after class. now Version 1 performance was worse than the theoretical baseline!

why is this possible?