1 of 16

What is essential for a winner DataHack project?

  • A great dataset
  • Original/Essential problem
  • A great team
  • All of the above

2 of 16

(2ב…)

Novel Multiple Choice Question Solver

Project Code: https://github.com/ilaif/datahack-2017

3 of 16

Answer when you know nothing.

  • A naive guess yields 25% success
  • Can we do better?

4 of 16

Dataset

  • 47,000 multiple choice questions (2 different datasets)
  • 4 answers per question
  • 1 answer is correct
  • Various subjects

5 of 16

Features

  • Answer / Question: word count, certainty measure, ...
  • Answer-to-(Answer/Question): similarity, ....

6 of 16

Solution

  • Not a classic classification problem
  • Anomaly Detection using Isolation Forest

Question

A

B

C

D

Question

B

A

C

D

Question

A

B

C

D

Question

D

A

B

C

​

Question

C

D

A

B

​

​

Question

B

C

D

A

​

​

​

Train

Test

(First answer is always the correct one)

(Find the shifted copy that is not an outlier)

7 of 16

Results

Model Accuracy: 28.6% !!!!!

Statistically significant (Hoeffding InEQ)

8 of 16

9 of 16

Myth #1: All of the above

Commonly thought as a likely choice

10 of 16

Myth #2: Avoid Certainty

Avoid words like: every, always, never, …

Prefer words like: maybe, sometimes, ...

11 of 16

Myth #3: Prefer simple words

For every answer, average frequency per word

12 of 16

Myth #4: Size matters

The longer the answer, the better

13 of 16

Myth #5: I’m on a roll

Knowing one answer helps predict the next

14 of 16

What is essential for a winner DataHack project?

  • A great dataset
  • Original/Essential problem
  • A great team
  • All of the above

15 of 16

What is essential for a winner DataHack project?

  • A great dataset
  • Original/Essential problem
  • A great team
  • All of the above

16 of 16

Thank You