Frontiers in Natural Language Processing Expert Responses
Organized by Herman Kamper, Sebastian Ruder, and Stephan Gouws at the Deep Learning Indaba 2018
You can find the slides of the session here.
1. What do you think are the three biggest open problems in NLP at the moment?
Task driven dialogue systems
For under resourced languages: Speech technologies and basic resources (eg stop word lists)
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
Word embeddings
3. What, if anything, has led the field in the wrong direction?
Sentiment analysis on labelled corpora - where do I even start!
4. What advice would you give a postgraduate student in NLP starting their project now?
Natural language labelling is expensive and genre specific. Focus on unsupervised and semi-supervised techniques.
1. What do you think are the three biggest open problems in NLP at the moment?
NLP for truly low-resource languages.
Estimating worst-case performance.
Life-long adaptation of low-level models for downstream tasks.
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
The broader idea of distant supervision, e.g., https://www.aclweb.org/anthology/P09-1113
3. What, if anything, has led the field in the wrong direction?
Anything new will temporarily lead the field in the wrong direction, I guess, but upon returning, we may nevertheless have pushed research horizons. I think there’s been way too many papers on word embeddings - and way too many hours spent on (at the time) poorly understood reinforcement learning algorithms - but it’s clear that both have inspired a lot of interesting research.
4. What advice would you give a postgraduate student in NLP starting their project now?
Mix up low-hanging fruits with questioning the current paradigms. It makes sense to get experience with the current fad, but you don’t win best paper awards adding layers to LSTMs.
1. What do you think are the three biggest open problems in NLP at the moment?
1. domain adaptation: we are sort of assuming that language is markedly different across domains as our models do not transfer well. But humans can read stuff in most domains (even unexposed ones) fairly well. So what do they abstract and alternatively ignore, and what knowledge are they transfering? How to get systems to similarly work well in multiple domain?
2. text and dialog generation: Neural network based methods have been great for end to end training of these systems which so far involved too many components in an error-prone pipeline. But while fluency is great in the output now, content is often flaky: the output is prone to misrepresentation, repetition, incompleteness, limited state (in dialog systems). We used to have the opposite problem before: content was good but bad fluency.
3. learning with limited data: Suddenly NLP is set towards very large datasets of everything--summaries, image captions, etc. And systems are data hungry. The problem with these very large datasets is that they are often approximate, because they are gathered using some heuristics from the web or so. For example, instead of an actual summary, you may take a gist of a webpage that possibly was meant for search engine indexing. Such data does not reflect the linguistically complex or interesting phenomena that were present in hand-annotated ones. So maybe we aren’t solving the interesting problems sometimes, but just getting neural-type systems to start to work for the task. Which is not a bad thing, but we would need to move on to more interesting stuff, and then data for complex phenomena will still be a problem, if we need a million examples to learn anything.
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
This is a very hard question. From the most recent, maybe sequence to sequence learning in the encoder-decoder framework. As a concept, it has been widely applicable in NLP.
3. What, if anything, has led the field in the wrong direction?
I guess NLP people are/were also interested in computational linguistics wherein you want to get some scientific understanding of language using computational techniques. This angle is somewhat less represented in NLP conferences now-a-days. Also, when we do most tasks, we no longer have a linguistic hypothesis or task-based story to tell. Models are fairly generic, there is a bag of tricks that is used for all, and not really thought out for individual tasks. And to be honest, I feel language is different from speech or vision. In speech or vision, the input is a signal or pixels and we never really knew how to compute features on them. They are not the natural form consumed by people. But language is different, the input to systems is the same form that people read on paper, it is the same orthographic foxrm that people can teach to other people. So eventually we should figure out good ways to add prior knowledge into models to get big boosts.
4. What advice would you give a postgraduate student in NLP starting their project now?
It has always been true that someone working on NLP needs a wide expertise and read from multiple disciplines. It remains the same although the emphasis on disciplines has changed a bit. Maybe one would read less psychology but more machine learning. My advice is to read widely early on. Also to have bigger goals than picking simple tasks and methods. Work on problems that actually matter, linguistically, socially, and economically.
1. What do you think are the three biggest open problems in NLP at the moment?
1. Transfer Learning. The ability to transfer models to new conditions, which includes learning under limited (or absence of) annotated resources. To be able to learn truly robust NLP models, for many more than the tiny amount of languages and domains which we can currently support. As a matter of fact most current approaches need abundant amounts of labeled training data, and work well only in benchmarking scenarios.
(which brings me to the next point)
2. Evaluation. This is often neglected, but it is an important point: we need both in-depth and thorough studies that shed light on why certain approaches do (or do not) work, as well as work that questions established evaluation measures. Especially now that more work is towards higher-level tasks (e.g. all the recent work in NLG) -- we need to reflect on current evaluation practices.
3. Introspection. Finding out what our models (and representations) capture, to take it as a starting point for building our next generation of NLU models, to get one step closer towards real understanding.
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
Hard to answer. If I had to pick one from the last half decade: sequence to sequence models (so work by Sutskever et al. and Cho). They advanced entire fields in the last few years (MT, speech, NLG). From the last decade: Collobert's et al.'s word2vec model.
3. What, if anything, has led the field in the wrong direction?
Learning datasets, rather than learning problems. Focusing too much on tiny unrepresentative datasets, missing out on the bigger picture.
4. What advice would you give a postgraduate student in NLP starting their project now?
- read a lot. both recent papers and classics. write literature surveys
- always keep in mind: what is the challenge/the most difficult question/the big picture behind the problem you are currently trying to solve?
- if you want to stay in academia: start writing grant applications early on
1. What do you think are the three biggest open problems in NLP at the moment?
* Transfer learning viz. using pre-trained models to build a task model f: X => Y that predicts values of Y given only a few training examples OR novel values of Y that were not part of the training data for the task (a.k.a. few-shot and zero-shot learning).
* Having longer goal/task oriented human-machine conversations that require real-world context and a knowledge base. Think for example of an e-coaching or behavioural nudging system, perhaps in health, that needs to understand, engage and build a good rapport with a user.
* Methods for gathering data and training language models for under resourced languages.
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
In general I would say deep learning (A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning by Colobert & Weston just won the ICML Test Of Time award), but more specifically I would say pre-trained language models starting from 'shallow' word embeddings to 'deeper' language models like OpenAI's Transformer. A nice post in this topic is https://thegradient.pub/nlp-imagenet/
3. What, if anything, has led the field in the wrong direction?
Perhaps the idea/hype that given enough conversational data a sequence to sequence model (or chatbot, etc) would by some magic become a conversational AGI.
4. What advice would you give a postgraduate student in NLP starting their project now?
I would say that students should look at 1) approaches to collect data for currently under resourced languages, 2) develop pre-trained models useful for transfer learning and 3) do research on enabling longer conversations with task oriented dialog systems, e-coaching systems, etc.
For 1&2 my concrete advice will be to look at say ELMo and the tasks that the pre-trained models have been evaluated on. Then think about putting the model and shared tasks in place for one or more of our local languages. For data one can start from sources such as the SADiLaR site and public news feeds, but students could also look at crowd sourcing more data.
For 3 I'm less sure of what needs to happen, but I would advise students to look at use cases and expected user journeys for e-coaching and behavioural nudging in health or education to see which parts of those systems could be automated to perhaps initially scale up how many people each coach/mentor can reach. The longer term goal should then be to automate more and more of the systems.
1. What do you think are the three biggest open problems in NLP at the moment?
Not open, but still needs a lot of improvement: Coreference resolution, Word sense disambiguation, Text Summarization. NLP is relatively successful in problems dealing with syntax, but it still needs a lot of improvement in semantics and pragmatics, for example, finding the meaning of word or word senses, determining scopes of quantifiers, finding referents of anaphora, relation of modifiers to nouns.
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
Deep learning for multi-level automatic feature representation learning, instead of relying on hand-crafted features.
3. What, if anything, has led the field in the wrong direction?
Chomskyan theories of linguistics instead of corpus linguistics that underlies the machine-learning approach to NLP.
4. What advice would you give a postgraduate student in NLP starting their project now?
Work on your machine learning skills, but also read/study wider in for example areas such as linguistics in the humanities.
1. What do you think are the three biggest open problems in NLP at the moment?
Learning to solve problems that don't have a lot of training data available to them.
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
Sorry, this sounds too much like a Vogue quiz.
3. What, if anything, has led the field in the wrong direction?
That important problems are correlated with problems for which a lot of data exists. I think that's getting it exactly the wrong way around. The important problems are ones where there is very little data.
4. What advice would you give a postgraduate student in NLP starting their project now?
Do hypothesis-driven research, don't follow trends.
1. What do you think are the three biggest open problems in NLP at the moment?
I wouldn't say these are necessarily the 'biggest', but I think some interesting research directions are:
1. Arguments / reasoning, sarcasm and humour. Practically, we are interested in the problem of automated comment moderation, or flagging toxic comments in online comment spaces. (This is an interesting paper on the topic.) More generally, the representation of longer texts, eg. sentences and documents, as opposed to words and characters, does not seem to be satisfactory yet.
2. Our reliance on biased training data, eg. this paper on gender bias in Word2Vec trained on Google News data.
3. Democratisation: NLP development in languages other than English, especially machine translation, but also tools like NER, stemming and parsing. Unsupervised NLP is a good direction to avoid the need for tagged data.
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
I think the most influential work within the field must be Word2Vec, based on its aptness, its elegance and the breadth of research it spawned. A special mention should probably be given to LSTMs, too, because they're brilliant.
3. What, if anything, has led the field in the wrong direction?
A linguist lecturer of mine, Henry Thompson, argued that while there has been immense progress in developing language applications that work well, based on large datasets, our efforts to understand how languages work on a fundamental level has stayed behind. Perhaps a better theoretical understanding would also provide better solutions to the tasks we are trying to mimic.
4. What advice would you give a postgraduate student in NLP starting their project now?
To me, a key realisation was that most of the approaches you learn about have caveats. As an engineering undergraduate, I was used to learning things as hard facts, but in NLP, the answers are less black and white, and most problems are only partially solved.
1. What do you think are the three biggest open problems in NLP at the moment?
Discourse, summarization, and language change
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
Possibly the word2vec paper, for its outsized influence on how we do NLP today. It's hard to find a paper that does not use embeddings, and often still word2vec.
3. What, if anything, has led the field in the wrong direction?
The construction of language as a pure engineering problem. It oversimplified one of the most complex human behaviors.
4. What advice would you give a postgraduate student in NLP starting their project now?
Remember that language comes from people, not machines.
1. What do you think are the three biggest open problems in NLP at the moment?
How to connect language with the perceptual world around us (e.g. in a robot)
How to connect language to all the knowledge in the web
How to evaluate language technology, especially e.g. dialogue
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
Since Elman 1991 is not in the last decade, I'll go for Collobert & Weston 2008ish (see also JMLR 2011).
3. What, if anything, has led the field in the wrong direction?
Unchallenged assumptions about what is a good representation of language based on speculation and subjective intuition rather than empirical evidence
A fear of anyone who is not 'a language person' trying to contribute to NLP (even tho many important ideas came from other research areas)
4. What advice would you give a postgraduate student in NLP starting their project now?
Understand psychology and the core problems of semantic cognition. Read e.g. Eleanor Rosch, Lakoff and Johnson etc. Go to CogSci. Understand machine learning. Go to NIPS.
Submit something terrible (or even good, if possible) to a workshop as soon as you can. You can't learn how to do these things without going through the process.
Don't worry about the quality of your earliest work. But do worry later of course.
Don't wait for your advisor to suggest what you should do, and don't let him/her tell you what you shouldn't.
1. What do you think are the three biggest open problems in NLP at the moment?
First, how do we construct better benchmarks that reveal the limitations of our methods at scale? One of our better benchmarks, MultiNLI, still has a lot of problems that are hard to correct (see https://arxiv.org/pdf/1803.02324.pdf).
Just looking at accuracies on popular benchmarks paints an overly optimistic picture of progress in building systems to read and write natural language.
We need a new generation of evaluation datasets and tasks that show whether our techniques actually generalize across the true variability of human language. I'd be much happier with benchmarks that understated progress than the current ones that seem to overstate it.
Second, how do we attach "steering wheels" to our generative models of text? How can we create models capable of following high level instructions about what to generate, or that can obey constraints about what to generate? I want models that can edit a piece of text based on simplified or restricted natural language instructions about what sort of high level change to make. How can we operationalize linguistic formalisms with natural language interactions? How can we construct interactive "proof" protocols that show that a statistical system captured everything important about a passage?
Third, how do we generate much more training data that is much more informative. Can we create language games that we can scale for things like editing or paraphrasing that naturally produce contrastive examples? Can we elicit more diverse linguistic examples from human annotators? Can we build massive training sets that bag of word embedding models really, completely fail to handle? Or, can we find better ways to achieve self-supervision with unannotated text?
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
The success of neural machine translation and its continued improvements. It is still brittle and makes too many errors, but nonetheless, amazing work!
3. What, if anything, has led the field in the wrong direction?
Over reliance on suboptimal benchmarks. Many of our models have performance only a bit better than strong bag of word embedding baselines paired with some simple tricks. Some of our datasets are poorly constructed or extremely limited (for example most paraphrase datasets).
4. What advice would you give a postgraduate student in NLP starting their project now?
Learn how to tune your models, learn how to make strong baselines, and learn how to build baselines that test particular hypotheses. Don't take any single paper too seriously, wait for its conclusions to show up more than once. In many cases, you can take even pretty solid published work and make much stronger baselines that give some more context on the results.
1. What do you think are the three biggest open problems in NLP at the moment?
Ranked in order of how open these problems are.
“True” natural language understanding. Our models great at doing surface pattern matching, but when a problem requires interpreting the local discourse and combining that with world knowledge that is not explicitly expressed in text, they often fail. For instance, “X gets in the car” requires X to first open the car door, but because this is so obvious to humans, it is rarely expressed in text. Similarly, there are topics considered taboo that we rarely talk about and thus a NLP model has a hard time learning about. So all of this is to say: we need to be better at combining text with other types of knowledge sources to achieve anything close to human natural language understanding.
Representing large contexts efficiently. Our current models are mostly based on recurrent neural networks, which cannot represent longer contexts well. One recent encouraging work in this direction I like is the NarrativeQA dataset for answering questions about books. The stream of work on graph-inspired RNNs is potentially promising, though has only seen modest improvements and has not been widely adopted due to them being much less straight-forward to train than a vanilla RNN.
Effective cross-domain transfer for low resource scenarios. This isn’t really a new problem and there is some recent work on it (unsupervised pre-training with ELMo, ULMFiT, Transformers; work on multi-task learning), but it’s only now starting to get traction in the NLP community. I believe we will see many innovative papers in this area in the next few years.
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
I would have to go with Collobert & Weston, A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning, https://ronan.collobert.com/pub/matos/2008_nlp_icml.pdf
Pretty much the seminal work on word embeddings
3. What, if anything, has led the field in the wrong direction?
I would say that the success of RNNs coupled with easily accessible deep learning frameworks has reduced the diversity of papers and also reduced the proportion of truly innovative papers at NLP conferences. (To some degree, this is part of an expected hype cycle and is already starting to die down.)
One resulting obsession of the field is beating the state of the art through “neural architecture search" on datasets released by well-known labs (not all of which are well designed), as opposed to aiming to get a thorough understanding of why models make behave in a certain way. Also, the focus on NLP engineering papers as opposed to other types of papers allowing for introspection. I really liked the initiative by the Coling PC to introduce different papers types in this regard (https://coling2018.org/paper-types/).
4. What advice would you give a postgraduate student in NLP starting their project now?
NLP is a very busy space these days and one of the things to be conscious of is not to be scooped. If I had to start a PhD project now, I would aim for a very ambitious and novel, yet impactful task and try to establish myself as the expert on that task.
Read a lot of papers and aim to understand them in depth, including the maths. Pay particular attention to how the papers relate to prior work, including work that is not explicitly cited. Don’t only read recent papers, but also pre-neural net papers and get a thorough grounding in different parts of machine learning (not only neural nets!). It can help a lot to establish a reading group with others.
Most interesting NLP papers combine existing research ideas in unexpected ways. If you have an additional background in another discipline (e.g. linguistics, cognitive science, sociology, image processing), perfect! Think about how to combine ideas from the two disciplines.
In addition to your main project, aim to spend 10-20% of your time on side projects and collaborations with others. You can learn a lot from collaborating with different people and it might help you out of a slump if your main project isn’t going so well.
1. What do you think are the three biggest open problems in NLP at the moment?
- Robust long-form text generation (including generation for dialogue systems).
- Adding reasoning abilities (including commonsense reasoning) to models.
- Finding the right representations and kinds of supervision required for learning to represent meaning.
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
I would have to say sequence to sequence models with attention (Bahdanau et al 2014), along with the broad application of bidirectional LSTMs, as those has transformed the way that most NLP tasks are approached.
3. What, if anything, has led the field in the wrong direction?
Most of the developments have been positive, but some potential issues are:
- Too much emphasis on tasks that require a lot of training data
- Not enough focus on the modelling biases required to address problems
- High performance on standard datasets might obscure the hard problems that still remain to be solved
4. What advice would you give a postgraduate student in NLP starting their project now?
Due to the success of relatively generic deep learning models in NLP there is a risk of not thinking broadly enough about how to approach problems. So it will be helpful to still have some knowledge of older NLP approaches, other machine learning methods and linguistics, even though they might not appear to be relevant anymore.
1. What do you think are the three biggest open problems in NLP at the moment?
1. Dealing with low-data settings (low-resource languages, dialects (including social media text "dialects"), domains, etc.). This is not a completely "open" problem in that there are already a lot of promising ideas out there; but we still don't have a universal solution to this universal problem.
2. Conversational agents that can keep track of dialogue and world context.
3. Representation and understanding of long documents (spoken or written).
3. Seamless integration of language-related modalities (speech, text, visual, other). Like #1, there are already good ideas out there, so one might argue how "open" it is.
(The careful reader will note that this is 4 problems, not 3, but two of them are tied for #3 :)
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
Word embeddings. These were introduced earlier, but important varieties (e.g. word2vec) were introduced in the past decade, and the use of word embeddings has become universal in the past decade. A close runner-up is RNNs, which again are not new but their near-universal use in speech and language tasks is.
3. What, if anything, has led the field in the wrong direction?
I can't think of an idea that was introduced and wasn't worth pursuing, and I can't say the field ever went "in the wrong direction".
4. What advice would you give a postgraduate student in NLP starting their project now?
Collaborate a lot. Do internships. Find multiple mentors and learn to weight their advice. Take courses on fundamental methods, even if they don't seem relevant right now. Learn the history of the field.
1. What do you think are the three biggest open problems in NLP at the moment?
I think the biggest open problems are all related to natural language understanding. (Even in generation tasks, I'd argue that the challenges all come down to the fact that computers do not understand what a piece of text means in the way a person does.) I think we should develop systems that read and understand text the way a person does, by forming a representation of the world of the text, with the agents, objects, settings, and the relationships, goals, desires, and beliefs of the agents, and everything else that humans create to understand a piece of text. Until we can do that, all of our progress is in improving our systems' ability to do pattern matching. Pattern matching can be very effective for developing products and improving people's lives, so I don't want to denigrate it, but I don't think that pattern matching alone will produce a machine that can understand text in the way a person does.
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
I would probably choose "Natural Language Processing (Almost) from Scratch" by Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa published in JMLR in 2011. It is based on a paper by Collobert and Weston from ICML 2008. This paper was way ahead of its time, introducing several methods to the NLP community that are commonplace today, like pretraining word embeddings with efficient ranking objectives on unlabeled data, using multi-task learning with neural nets for NLP, using a CRF "layer" atop a neural network used to produce features, and using convolutional networks for NLP, among others.
3. What, if anything, has led the field in the wrong direction?
I think there has been a heavy focus on problem settings that are downright artificial. There are several settings I'm thinking of here. One is traditional supervised learning in which the test data is assumed to come from the same distribution as the training data. In the real world, the test data is almost never from the same distribution as the training data. At a minimum, the real-world test data is separated in time from the training data, sometimes by decades! For example, it's very common to use a POS tagger trained on the Wall Street Journal portion of the Penn Treebank and use it to label modern newswire. We should have been emphasizing out-of-domain, temporally-shifted, and other transfer settings starting in the 90s. Much more artificial are the many unsupervised learning settings that many researchers (myself included) have devoted a lot of energy to over the years.
To summarize: traditional unsupervised learning and traditional supervised learning are both unrealistic, so I've been glad to see NLP researchers focus recently on hybrid settings, whether they're described as semi-supervised, weakly-supervised, "Wiki-ly" supervised, distantly-supervised, lightly-supervised, minimally-supervised, or something else. :) Using pretrained word embeddings is one such hybrid setting, and it's nice to see that this is now standard.
4. What advice would you give a postgraduate student in NLP starting their project now?
Don't be afraid to do something new, ambitious, and creative, even if it may involve creating a new dataset, task, or evaluation. Often these seem like high-risk directions, but I think they are lower risk than trying to beat the state of the art on a widely-studied task. If you try something highly ambitious and creative, even if it feels like a failure to you because it doesn't live up to your expectations, you will likely end up with a lot of interesting (and therefore publishable) material.
1. What do you think are the three biggest open problems in NLP at the moment?
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
3. What, if anything, has led the field in the wrong direction?
4. What advice would you give a postgraduate student in NLP starting their project now?
my answer to all these questions: "people should do whatever they want".
please take this for the serious and deep answer that it is!
1. What do you think are the three biggest open problems in NLP at the moment?
perhaps the biggest question in NLP/MT is when and in which direction the next leap would happen. NLP/MT has seen two big changes during the past half century. the first moment of change was in early-mid 90's when statistical approaches have become dominant, pushing the field away from rule-based approaches. the second moment was in earlier this decade (2012-2015) when deep learning had become a dominant approach. it's not only that each leap led to the empirical improvement, but also had shaken our (presumed) understanding of the field and problems we were tackling. what would be our next leap? i have my guess, and everyone has their own guess, but my true conviction is that the next leap will start by a new person/group, as has it been the case for the past two occasions...
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
it's really difficult to choose one, but i must say collobert & weston (2008) and its follow-up journal paper. in these papers, collobert & weston proposed and demonstrated the effectiveness of a neural net based approach to various problems in NLP that was a radical departure from whatever others were doing.
3. What, if anything, has led the field in the wrong direction?
i don't think the field has been led in any wrong direction. i strongly believe that science does not always advance forward but it's rather a guided random walk generally toward better understanding. what may seem like a wrong direction ysterday may turn out to have been a right direction today or tomorrow. for instance, many, including myself, worked on layer-wise unsupervised pretraining for training a deep net between 2006 and 2012, which looked extremely promising back then, but is not being pursued (at least actively) anymore. was it a wrong direction? i certainly don't think so, as we learned a lot from those years (ReLU, amortized inference and so on are relics from that era.) similarly in NLP, there were impossibly many papers in 2014 and 2015 on applying word2vec to anything people could find, most of which have already been forgotten. was it a wrong direction? probably not. we just needed this period to make another step further. i believe a better question would be in this "guided random walk" of science, what should the guide be?
4. What advice would you give a postgraduate student in NLP starting their project now?
i believe scientific pursuit is meant to be full of failures. 99 out of 100 ideas you come up with are supposed to fail. if every idea works out, it's either (a) you're not ambitious enough, (b) you're subconciously cheating yourself, or (c) you're a genius, the last of which i heard happens only once every century or so. so, don't despair!
1. What do you think are the three biggest open problems in NLP at the moment?
-> making things work in low resource environments (be it language or task as the bottleneck)
-> language *understanding*. We've become incredibly good at transforming one seq into another by looking at huge amounts of sequence pairs -- but those models don't have any useful intermediate (meaning) representation. A related practical issue is interpretability: the best models we have are hard to interpret and it is hard to understand and justify the predictions they make.
-> building flexible systems that can cope e.g., with change in input data distribution, or solve multiple related tasks (this isn't NLP specific but crucial for deployed systems in general in my opinion)
2. What would you say is the most influential work in NLP in the last decade, if
you had to pick just one?
-> The most influential is pretty clearly the development in deep learning (most notably seq2seq for tasks like speech recognition, MT or summarisation). I'm less sure if this intellectually advanded the field too much, though
3. What, if anything, has led the field in the wrong direction?
-> One slight embarrassment were recent misdesigned benchmarks. E.g., humans were asked to produce large amounts of questions given an image (like 'how many horses are in this image?') and systems were trained to do visual QA. Turned out that these systems do not learn to parse the image, but just idiosyncrasies of the dataset (e.g., most 'how many?' questions have the answer '2', so the systems cannot generalize to an image with 3 horses). Although in general I think there's something to be learnt from every dead end
4. What advice would you give a postgraduate student in NLP starting their project now?
-> talk to and learn from your colleagues, especially your peers, from day 1. Embrace the multi-disciplinarity of NLP!
1. What do you think are the three biggest open problems in NLP at the moment?
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
3. What, if anything, has led the field in the wrong direction?
4. What advice would you give a postgraduate student in NLP starting their project now?
1. What do you think are the three biggest open problems in NLP at the moment?
Language understanding, all current neural models are just remembering patterns but they can't really understand the language.
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
Seq-to-seq models for MT
4. What advice would you give a postgraduate student in NLP starting their project now?
Keep questioning the evaluation metrics and the quality of datasets that you are working on.
1. What do you think are the three biggest open problems in NLP at the moment?
* Inferring what's *implicit* in a text (implicit arguments, discourse relations, etc.)
* Understanding intentions of authors (i.e., not just what is written or said but also *why*)
* Scaling NLP methods to more domains (and languages) and wider application scenarios
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
Most influential on me: <something random from linguistics>. Most influential on the field as a whole: probably word2vec.
3. What, if anything, has led the field in the wrong direction?
I don’t think the field as a whole is moving in a single wrong direction. It's just individuals moving in (subjectively) better/worse directions. I believe that everyone has their own legitimate reasons for what they are doing, even if it’s just to work on low hanging fruits.
4. What advice would you give a postgraduate student in NLP starting their project now?
Start your project with an interesting research question.
1. What do you think are the three biggest open problems in NLP at the moment?
Demonstrate natural language understanding and generation capabilities. Things that are obvious to humans, for example:
Question answering (Squad problem).
Achieve human translation accuracy. Machine translation including language subtleties. Japanese people think that Google translate is a joke!
Dialog.
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
Seq2seq! http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf
3. What, if anything, has led the field in the wrong direction?
I don't think there is anything like that. We can learn from "wrong" directions and "correct" directions, if such a thing even exists.
4. What advice would you give a postgraduate student in NLP starting their project now?
1. Try to read papers/blogs every single day.
2. In order to have a deep understanding of a problem, it is sometimes better to code/reimplement than using available open source solutions.
1. What do you think are the three biggest open problems in NLP at the moment?
1) Perhaps the biggest problem is to properly define the problems themselves. And by properly defining a problem, I mean building datasets and evaluation procedures that are appropriate to measure our progress towards concrete goals. Things would be easier if we could reduce everything to Kaggle style competitions!
2) Low-resource languages. English is not the only language in the world!
3) People have tried to use linguistic knowledge to understand and improve our models, which is great, but I would love to see more of the opposite: can our models help us understand how language works?
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
"Distributed representations of words and phrases and their compositionality" by Mikolov et al.
3. What, if anything, has led the field in the wrong direction?
This is research, not engineering, so only time will show if we are going in the wrong direction or not! For that same reason, something that I don't like is when people make categorical claims on how our solutions need to be, discrediting people with different views (e.g. NLP cannot be solved without linguistics, so if your approach ignores linguistics, it is garbage to me).
4. What advice would you give a postgraduate student in NLP starting their project now?
Be open, listen and learn form other people but, at the same time, follow your intuition and get enough freedom to work on your own ideas. It is not only more rewarding, but you are also way more efficient when you do. And, being cynical, if all your research ideas happen to be terrible, you have no future on this, anyway, so the sooner you realize the better :P
1. What do you think are the three biggest open problems in NLP at the moment?
Common sense reasoning that combines logical and statistical information
Multitask learning
Better real dialogue systems
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
Collobert and Weston
3. What, if anything, has led the field in the wrong direction?
The unclear separation between linguistics and nlp
4. What advice would you give a postgraduate student in NLP starting their project now?
The next Frontier for impact is multitask learning.
1. What do you think are the three biggest open problems in NLP at the moment?
Learning with small data (transfer, multitask, meta-learning, prior knowledge, etc). In many ways, this subsumes everything else (e.g., common sense would be easy if we had enough common sense examples for task X).
Multi-document reasoning: how can we aggregate information from unstructured sources and infer things we don’t yet know? How should this information be represented? How can this be learnt end-to-end? etc.
Integrating/Extracting Common sense for/from language processing.
2. What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
Natural language understanding (almost) from scratch
3. What, if anything, has led the field in the wrong direction?
I’m not sure if I know what *led* the field into wrong directions, but the following *are* some wrong directions
Ignoring previous work in NLP (end elsewhere) before the DL revolution
Benchmark/Leaderboard chasing and related “architecture hacking”
What led to it is hard for me to say. Surely reviewer biases, but that’s chicken-and-egg. Note that this is nothing new. I think the collective memory loss is partly a consequence of the reduced entry barrier for outsiders that came through DL (which is a good thing, of course).
4. What advice would you give a postgraduate student in NLP starting their project now?
I omit any general PhD/research advice and focus what I think is NLP specific
Look at data, problems, and as much "language language language” as possible. (Respect “language”---see Yoav’s rant). Look at the predictions of your model, do error analysis.
Learn ML *and* (some level of) Linguistics.
1. What do you think are the three biggest open problems in NLP at the moment?
Grounded language learning, i.e., jointly learning a world model and how to refer to it in natural language
Merging language understanding and reasoning within a deep learning framework
Commonsense understanding, which can only be solved along with the above two
3. What, if anything, has led the field in the wrong direction?
Greed. We are seeking short-term results and rewards, so we try to exploit the data at our disposal, like vast quantities of text, hoping we can train models which can understand and generate language intelligently, but it cannot happen - whatever the neural architecture trickery we manage to design - if we do not also solve the harder problem of building world models, i.e., models which understand how our world works, with humans and other agents in it. We need instead to bite the bullet and go for the bigger prize of solving AI altogether with NLP, not just NLP in isolation.
4. What advice would you give a postgraduate student in NLP starting their project now?
Be ambitious. Do not limit yourself to reading NLP papers. Read a lot of machine learning, deep learning, reinforcement learning papers. A PhD is a great time in one's life to go for a big goal, and even small steps towards that will be valued.