Project list - MVA SLP- 2021

	A	B	C	D	E	F	G
1	project id	project field	project area	title of paper	link to paper	Github	Task

2	PROJECT 1 / PROJECT 2 (cf. "task")	Speech processing	Self-supervised pretraining for low resource languages	wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations / Unsupervised Pretraining Transfers well Across Languages.	https://arxiv.org/abs/2002.02848 https://arxiv.org/abs/2006.13979 https://arxiv.org/abs/2006.11477	https://github.com/pytorch/fairseq/blob/master/examples/wav2vec/README.md https://github.com/facebookresearch/CPC_audio	Self-supervised pretraining (CPC, Wav2Vec, etc) on large quantity of audio has proven very succesful in learn speech features that achieve very good results on several speech tasks with minimal amounts of labelled data (10 min-10h). (see blogpost: https://maelfabien.github.io/machinelearning/wav2vec/#g-results) The features learned in one language seem to even be good enough on other untrained languages. Two projects are possible here. PROJECT 1. Using CPC or Wav2vec pretrained features or models and implementing phone recognition with a small network trained with CTC in a new language with only 10min-1h-10h of training data. The project can just use frozen features, or fine tune the model, or compare both. Small datasets for a new language are available in the Mozilla common voice project. PROJECT 2. Even through pretraining works well, we still do not undestand why. The aim of this project, it to explore how a pretrained wav2vec encodes speech, by training linear classifiers on frozen features extracted from each layers, and plot the resulting PER layer by layer. Speaker classifiers, or prediction of pitch can also be trained to undestand how other non-phonetic acoustic informations are encoded.
3	PROJECT 3	Computational linguistics, machine learning	Language emergence in neural agents	Anti-efficient encoding in emergent communication / Lazimpa: Lazy and impatient agents learn to communicate efficiently	https://papers.nips.cc/paper/2019/file/31ca0ca71184bbdb3de7b20a51e88e90-Paper.pdf https://www.aclweb.org/anthology/2020.conll-1.26/	https://github.com/facebookresearch/EGG https://github.com/MathieuRita/Lazimpa	The aim of these papers is to study how universal properties of human languages (here efficient coding of the transmitted information) can emerge (or not) in simple communicative agents depending on the architecture and inductinve biases of these agents. The aim of the project is to reproduce one experiment of one of these two papers, propose and motivate a new experiment and test it.
4	PROJECT 4	NLP	Evaluation of Machine Translation	Unsupervised Quality Estimation for Neural Machine Translation	https://arxiv.org/pdf/2005.10608.pdf	https://github.com/pytorch/fairseq/blob/master/examples/unsupervised_quality_estimation/README.md	Quality Estimation (QE) is an important component in making Machine Translation (MT) useful in real-world applications, as it is aimed to inform the user on the quality of the MT output at test time. In this paper, the authors devise an unsupervised approach to QE where no training or access to additional resources besides the MT system itself is required, while achieving a level of correlation with human judgment which is almost as good as supervised approaches. The goal of the project is to reproduce at least some of the results described in the paper, and to suggest, motivate and obtain some new results. Of course, modifying, maybe even reimplementing bits of code would be a plus. Note that re-training MT models is probably beyond the scope of such a project, because of the computational and time cost it requires.
5	PROJECT 5	NLP	Neural Contextual Language Models and their use for downstream NLP tasks	CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters	https://arxiv.org/abs/2010.10392	https://github.com/helboukkouri/character-bert	CharacterBERT is a language model architecture whose goal is to combine the character-level awareness of the first layer of language models such as ELMo with the modelling power of Transformer architectures such as used in BERT, ROBERTa, CamemBERT and other models. The goal of the project is to become familiar with these language models and to suggest, motivate and carry out experiments that demonstrate how CharacterBERT can be used and/or how it compares with BERT on at least one NLP task. Of course, modifying, maybe even reimplementing bits of code would be a plus, as would be any model re-training.
6	PROJECT 6	NLP	Language models for classification tasks	RoBERTa: A Robustly Optimized BERT Pretraining Approach / CamemBERT: a Tasty French Language Model	https://arxiv.org/abs/1907.11692 https://www.aclweb.org/anthology/2020.acl-main.645/	https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.md https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.custom_classification.md https://camembert-model.fr	ROBERTa (trained on English data) and CamemBERT (a ROBERTa-like model trained on French data) are contextual language models based on a Transformer architecture. The goal of this project is to become familiar with one of these models (or both) and to use them in order to develop a classification model, for instance by reproducing, adapting or getting inspiration from the ROBERTa-based IMDB classification task (cf. the second of the three GitHub repositories cited).
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100