Quality estimation
* Required
What is your pennkey username?
*
Your answer
If you are working with a partner, what is your partner's pennkey username?
Your answer
Comparing aggregation methods
What was the agreement (kappa) between the labels assigned to the urls by the majority vote algorithm and the weighted vote algorithm?
It is okay if the algorithms give the same labels (kappa = 1).
Your answer
What was the agreement (kappa) between the labels assigned to the urls by the majority vote algorithm and Crowdflower's algorithm?
It is okay if the algorithms give the same labels (kappa = 1).
Your answer
What was the agreement (kappa) between the labels assigned to the urls by the weighted vote algorithm and Crowdflower's algorithm?
It is okay if the algorithms give the same labels (kappa = 1).
Your answer
What was the average worker quality according to the majority vote algorithm?
Your answer
What was the average worker quality according to the weighted vote algorithm?
Your answer
What was the average worker quality according to Crowdflower's algorithm?
Your answer
What was the correlation coefficient (tau) between worker qualities assigned by the majority vote algorithm and the weighted vote algorithm?
Your answer
What was the correlation coefficient (tau) between worker qualities assigned by the majority vote algorithm and Crowdflower's algorithm?
Your answer
What was the correlation coefficient (tau) between worker qualities assigned by the weighted vote algorithm and Crowdflower's algorithm?
Your answer
If you wanted to reverseengineer Crowdflower's algorithm, what other information could you consider incorporating?
Your answer
The EM Algorithm
Fill in the worker qualities and data labels after each iteration. Hint: After iteration 0, your worker qualities should all be uniform and the data labels should be computed by simple majority vote. At the start of iteration 1, you should recompute your worker qualities.
Worker qualities after iteration 0
One per line, so worker1 is first and worker5 is last.
Your answer
URL lables after iteration 0
One per line, in the order sunnyfun, sexmission, google, youporn, yahoo)
Your answer
Worker qualities after iteration 1
One per line, so worker1 is first and worker5 is last.
Your answer
URL lables after iteration 1
One per line, in the order sunnyfun, sexmission, google, youporn, yahoo)
Your answer
Worker qualities after iteration 2
One per line, so worker1 is first and worker5 is last.
Your answer
URL lables after iteration 2
One per line, in the order sunnyfun, sexmission, google, youporn, yahoo)
Your answer
Submit
Never submit passwords through Google Forms.
This content is neither created nor endorsed by Google.
Report Abuse

Terms of Service

Additional Terms
Forms