Quality estimation
If you are working with a partner, what is your partner's pennkey username?
Comparing aggregation methods
What was the agreement (kappa) between the labels assigned to the urls by the majority vote algorithm and the weighted vote algorithm?
It is okay if the algorithms give the same labels (kappa = 1).
What was the agreement (kappa) between the labels assigned to the urls by the majority vote algorithm and Crowdflower's algorithm?
It is okay if the algorithms give the same labels (kappa = 1).
What was the agreement (kappa) between the labels assigned to the urls by the weighted vote algorithm and Crowdflower's algorithm?
It is okay if the algorithms give the same labels (kappa = 1).
What was the average worker quality according to the majority vote algorithm?
What was the average worker quality according to the weighted vote algorithm?
What was the average worker quality according to Crowdflower's algorithm?
What was the correlation coefficient (tau) between worker qualities assigned by the majority vote algorithm and the weighted vote algorithm?
What was the correlation coefficient (tau) between worker qualities assigned by the majority vote algorithm and Crowdflower's algorithm?
What was the correlation coefficient (tau) between worker qualities assigned by the weighted vote algorithm and Crowdflower's algorithm?
If you wanted to reverse-engineer Crowdflower's algorithm, what other information could you consider incorporating?
The EM Algorithm
Fill in the worker qualities and data labels after each iteration. Hint: After iteration 0, your worker qualities should all be uniform and the data labels should be computed by simple majority vote. At the start of iteration 1, you should recompute your worker qualities.
Worker qualities after iteration 0
One per line, so worker1 is first and worker5 is last.
URL lables after iteration 0
One per line, in the order sunnyfun, sex-mission, google, youporn, yahoo)
Worker qualities after iteration 1
One per line, so worker1 is first and worker5 is last.