Minutes from the brainstorming session on
Crowdsourcing Games (and Educational Apps) for MWEs

WG1 session at the 7th PARSEME meeting in Dubrovnik, 27 September 2016

by Federico Sangati

Crowdsourcing games are becoming more and more popular. They are a way to engage people outside academia in research-related topics, and a way to collect large amount of annotated data in an affordable way.

We agreed it would be interesting to gather a community of NLP researchers, call it MWE-GAMERS team, interested to pursue this direction further after Parseme.

Apart from implementation efforts, the main obstacle to produce a successful crowdsourcing systems is the ability to reach the large public. I believe that the large network Parseme people have consolidated in the past years constitutes a perfect base to test any such system in many different languages and after it is ready, spread it around across all European countries involved in the Cost action (and beyond).

When it comes to NLP-crowdsourcing games, there are already many being developed. Just to name a few examples:

  • duolingo.com - developed by Luis van Ahn (who has developped many successful crowdsourcing platforms)
  • zombilingo.org, about dependencies in a sentence (only french) developed by Bruno Guillaume (Yannick direct contact)
  • jeuxdemots.org, lexicon networks - semantics (temporarily unavailable) - developed by Mathieu Lafourcade (Yannick direct contact)
  • clozemaster.com - learn languages (similar to duolingo) by filling the blank (pointed by simon)
  • phrase detective - anaphora resolution - developed by Massimo Poesio's team (Doug's direct contact)
  • wordrobe - pos / named entity - developed by Johan Bos team - Groningen University

In addition to these, Simon showed us a new mwe-related game his team has developed recently: igra-besed.si (game of words in Slovenian). It is based on specific collocation patterns (e.g., ADJ - NOUN) extracted from sketchengine API: a user sees the noun and has to guess the adjective that goes along with that noun. It also works in "competition mode", letting two players play side by side in real time. It would be interesting to extend this game to other languages. This would not require much manual effort since all the exercises are generated automatically.

In addition, I've presented a prototype of another mwe-related game: mwe.herokuapp.com, developed together with Angelo Basile. It is based on a popular TV Italian game show (la ghigliottina), also popular in other countries. You are given 5 words and you have to guess a sixth. Take these for example: whole - extended - close - large - nuclear. The secret word would be...family. In fact it is the word which co-occurs with all the 5 given words.

At the end of the brainstorming meeting, we agreed it would be very useful to have a survey (e.g., on corpora list) to gather a more complete list of NLP crowdsourcing games currently available. It will serve the purpose of comparing the differences in technical implementation choices and consolidating the NLP/MWE-GAMERS team.

Finally, a somewhat related direction to crowdsourcing games that came up is educational applications. Students in school already have to do many NLP/MWE related exercises as part of their normal curriculum and it‚Äôs not necessary to come up with an extremely fun and exciting game, as long as it‚Äôs more interesting (e.g., interactive) than the typical pencil and paper version. This also relates to Marko's project www.hr4eu.eu.

Many thanks to Manfred and Carla to have reserved some space for this brainstorming during WG1 session, I think it was very fruitful not only for us but for other teams as well.

Feel free to distribute this email around to other people who you think might be interested in this initiative and let's discuss this further (please reply to all with your comments).