NLP related GWAP and CrowdSourcing Systems
The version of the browser you are using is no longer supported. Please upgrade to a supported browser.Dismiss

View only
NameActiveShort DescriptionLanguagesURLReferencesNotes
Open mind word expertWord sense taggingEnglish-- T. Chklovski and R. Mihalcea. Building a sense tagged corpus with open mind word expert. In Proceed- ings of the ACL-02 Workshop on Word Sense Disambiguation: Recent Successes and Future Directions - Volume 8, WSD ’02, 2002.
- R. Mihalcea. Open mind word expert: Creating large annotated data collections with web users’ help. In In Proceedings of 4th International Workshop on Linguistically Interpreted Corpora (LINC-03), 2003.
1001 ParaphrasesAims to collect paraphrase varians in order to train a machine translation system.
English-- T. Chklovski. Collecting paraphrase corpora from volunteer contributors. In Proceedings of the 3rd International Conference on Knowledge Capture, K-CAP ’05, 2005.
- T. Chklovski. 1001 paraphrases: Incenting responsible contributions in collecting paraphrases from volunteers. In Knowledge Collection from Volunteer Contributors, Papers from the 2005 AAAI Spring Symposium, Technical Report SS-05-03, Stanford, California, USA, March 21-23, 2005.
- see
Free Association
Set of 2 player GWAP. Collection of associations between terms and their D. Vickrey, A. Bronzan, W. Choi, A. Kumar, J. Turner-Maier, A. Wang, and D. Koller. Online word games for semantic data collection. In Proceedings of the Conference on Empirical Methods in Nat- ural Language Processing, EMNLP ’08, 2008.
VerbosityA GWAP to gather common sense knowledge about words (connected to ConceptNet5), e.g., spinach is a type of vegetable, is green, is contained in sandwiches.English Speer, R., Havasi, C., and Surana, H. 2010. Using verbosity: Common sense data from games with a purpose. In Proceedings of the 23rd International Florida Artificial Intelligence Research Society Conference (FLAIRS).
DuoLingoA platform to teach a foreign language to people and encourage them to translate the Web into their native languages.Many L. von Ahn. Duolingo: Learn a language for free while helping to translate the web. In Proceedings of the 2013 International Conference on Intelligent User Interfaces, IUI ’13, 2013.
Sentiment QuizCollect users rating about the sentiment polarity of given sentences.English W. Rafelsberger and A. Scharl. Games with a purpose for social networking platforms. In Proceedings of the 20th ACM Conference on Hypertext and Hypermedia, HT ’09, 2009.
OntoGamesProject with multile games forbuilding ontologies, annotating data, and aligning multiple ontologies.English-- K. Siorpaes and M. Hepp. Games with a purpose for the semantic web. IEEE Intelligent Systems, 2008.
- K. Siorpaes and M. Hepp. OntoGame: Weaving the Semantic Web by Online Games, pages 751–766. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008.
Phrase DetectiveA GWAP for anaphora resolution: identify relationships between pronouns and the elements in the text they refer to.English J. Chamberlain, M. Poesio, and U. Kruschwitz. Phrase detectives - a web-based collaborative annotation game. In In Proceedings of I-Semantics, 2008.
- J. Chamberlain, M. Poesio, and U. Kruschwitz. A new life for a dead parrot: Incentive structures in the phrase detectives game. In In Proceedings of the Webcentives Workshop at WWW’09, 2009.
PlayCorefGWAP for anaphora resolution similar to Phrase DetectiveCzech, English Hladka, B., Mirovsky, J., Schlesinger, P. Designing a Language Game for Collecting Coreference Annotation. In Proceedings of the Third Linguistic Annotation Workshop, ACL-IJCNLP 2009, pp. 52-55, Suntec, Singapore, 2009.
- Hladká Barbora, Mírovský Jiří, Kohout Jan. An attractive game with the document: (im)possible?. In: The Prague Bulletin of Mathematical Linguistics, No. 96, pp. 5-26, Aug 2011.
WordrobeGWAP consisting in a large set of multiple-choice questions on part of speech tagging, named entity recognition, word senses disambiguation to annotate the Groningen Meaning Bank. The players need to answer these questions, scoring points depending on the agreement with fellow players.English (one of the games, Pointers, has also a Finnish implementation) N. Venhuizen, V. Basile, K. Evang, and J. Bos. Gamification for word sense labeling. In Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013). 2013.
EmojiWorldBotA multilingual emoji tagging game for people to contribute to the expansion of existing dictionaries or the creation of new ones for any additional language.
Many J. Monti, F. Sangati, F. Chiusaroli, M. Benjamin, and S. Mansour. Emojitalianobot and EmojiWorldBot - new online tools and digital environments for translation into emoji. In Proceedings of Third Italian Con- ference on Computational Linguistics (CLiC-it 2016). 2016.
JeuxdemotsJeuxDeMots (JDM) is a two player GWAP, launched in September 2007, that aims to build a large lexico-semantic network (for French) composed of terms (nodes) and typed relations (links between nodes). It contains terms and possible refinements in the same spirit as WordNet, although it is organised as decision trees. There are more than 100 different relation types, the occurrences of which are weighted. In January 2017 (after 10 years from the game launch), more than 60 million relations between over 1 million words have been collected.French (some prototypes have been developped in other languages, like English, Spanish, Japanese, Thai, Arabic M. Lafourcade, N. Le Brun, and A. Joubert (2015) Games with a Purpose
- M. Lafourcade, A. Joubert (2015) TOTAKI: A Help for Lexical Access on
the TOT Problem. In Gala, N., Rapp, R. et Bel-Enguix, G. éds. (2015),
Language Production, Cognition, and the Lexicon. Festschrift in honor of
Michael Zock. Series Text, Speech and Language Technology XI.
Dordrecht, Springer.
- J. Chamberlain, K. Fort, U. Kruschwitz, M. Lafourcade and M. Poesio
(2013) Using Games to Create Language Resources: Successes and
Limitations of the Approach. Theory and Applications of Natural Language
Processing. Gurevych, Iryna; Kim, Jungi (Eds.), Springer.
- M. Lafourcade, (2007) Making people play for Lexical Acquisition. In
Proc. SNLP 2007, 7th Symposium on Natural Language Processing.
- developed by Mathieu Lafourcade
- not clear if the purpose is to gather NLP data
ZombiLingoFinding syntactic relations (e.g., subject) in a sentence.French Bruno Guillaume, Karën Fort, Nicolas Lefebvre. Crowdsourcing Complex Language Resources: Playing to Annotate Dependency Syntax. Proceedings of the International Conference on Computational Linguistics (COLING), Osaka, Japan, 2016
- Karën Fort, Bruno Guillaume and Hadrien Chastant. Creating Zombilingo, a Game With A Purpose for dependency syntax annotation. Proceedings of the Gamification for Information Retrieval (GamifIR'14) Workshop, Amsterdam, Netherlands, April 2014
- developed by Bruno Guillaume
- not clear if the purpose is to gather NLP data
Puzzle Racer
2 video-game like GWAP where players in
1) maneuver a racer around obstacles and enemies in order to annotate concepts with associated pictures,
2) disambiguate the meaning of a word by blowing up relevant pictures to a target word like in fruit ninja
English, French, Spanish- D. Jurgens and R. Navigli. It’s all fun and games until someone annotates: Video games with a purpose for linguistic annotation. Transactions of the Association for Computational Linguistics, 2:449–464, 2014.
- The game is currently offline but may be brought back online soon.
The Knowledge Towers
1) A video-game like GWAP, reminiscent of Contra for the NES, where players destroy zombies threatening to invade a base. Players use a call-and-response mechanism to detect infected humans, which generates linguistic annotations for a variety of tasks, such as word similarity, antonymy, and relation extraction.
2) A video-game like GWAP reminiscent of Zelda for the SNES, where players navigate through a tower-like maze killing zombies and collecting treasure. Players are rewarded for collecting pictures related to a specific concept that changes, which results in annotating and validating a large knowledge base of concepts with pictures (BabelNet).
English, Italian D. Vannella, D. Jurgens, D. Scarfini, D. Toscani, and R. Navigli. Validating and extending semantic knowledge bases using video games with a purpose. In Proceedings of the 52nd Annual Meeting of the ACL 2014.
- The URL is currently down due to technical issues but should be brought back up soon.
XtribeWeb platform designed for scientific gaming and social computation including "Creastoria", an app for writing stories collaboratively.English, Italian S. Caminiti, C. Cicali, P. Gravino, V. Loreto , V.D.P. Servedio, A. S^rbu and F. Tria, XTribe: a web-based social computation platform, Proceedings of the Third International Conference on Social Computing and Its Applications (SCA), 2013.
SmallWordlOfWordsCollections of words associationsMany De Deyne, S., Navarro, D., Storms, G. (2013). Better explanations of lexical and semantic cognition using networks derived from continued rather than single word associations. Behavior Research Methods, 45 (2), 480-498.
PhraTrisGWAP for the annotation of sentences with syntactic dependencies.
Ingra-besedBased on specific collocation patterns (e.g., ADJ - NOUN). A user sees the noun and has to guess the adjective that goes along with
ClozemasterLanguage learning game based on filling the blank exercise.Many not clear if the purpose is to gather NLP data
BisamePart of speech taggingAlsatian under developed by a student of Karen Fort
Zooniverse4 Literature Digitization and Tagging Related Projects:
- Shakespeare's World
- Measuring the Anzacs
- Science Gossip
- Operation War Diary
QlaaraOnline gamified psycholinguistic experimentsEnglish
Tile AttackTile AttackEnglish
KrikKrikGuadeloupean Creole
LingoBoingoPlatform with multiple Word Languages Game (jeaux de mots, phrase detective, title attack, zombiligo)French/English