Advances in ML: Theory meets practice

Workshop at the Applied Machine Learning Days 2019
January 27, 2019, Lausanne, Switzerland




Welcome Remarks

13:35 - 14:15

Lénaïc Chiza (CNRS)t: Theory of neural networks training: challenges and recent results

14:15 - 14:55

Claudiu Musat (Swisscom): Theory Vs. Practice – It’s a Data Problem

14:55 - 15h10

Short break

15:10 - 15:50

Aaron Klein (Uni Freiburg): Bayesian Hyperparameter Optimization for Automated Machine Learning

15:50 - 16:30

Armand Joulin (Facebook): Multilingual word alignment


Lénaïc Chizat: Theory of neural networks training: challenges and recent results

CNRS researcher in Orsay (France)

The current successes achieved by neural networks are mostly driven by experimental exploration of various architectures, pipelines, and hyper-parameters, motivated by intuition rather than precise theories. Focusing on the optimization/training aspect, we will see in this talk why pushing theory forward is challenging, but also why it matters and key insights it may lead to. Along the way, we will present some recent results on the role of over-parameterization, on the phenomenon of "lazy training" and on training neural networks with a single hidden layer.

Claudiu Musat: Theory Vs. Practice – It’s a Data Problem

Director of Research, Data, Analytics & AI, Swisscom

Focusing on the creation of dialogue systems, the Swisscom ML research team has been often asked to improve systems that are currently in use. To an ML person, this is a dream case, as the starting expectation is that any ML system should beat hand crafted rules. Moreover, after framing the problem correctly, we find that it is actually well studied and that the solutions abound. The task is easy. In theory.

We find that the data available at the start of projects is not there, not in the right format or simply not enough. In this talk I will discuss possible solutions to the low data availability and showcase several problems where traditional approaches fail. In practice.

Aaron Klein: Bayesian Hyperparameter Optimization for Automated Machine Learning

Phd student, Machine Learning Lab (Frank Hutter), University of Freiburg

Machine learning has recently achieved great successes in a wide range of practical applications, but the performance of the most prominent methods depends more strongly than ever on the correct setting of many internal hyperparameters. The best-performing models for many modern applications are getting ever larger and thus more computationally expensive to train, but at the same time both researchers and practitioners desire to set as many hyperparameters automatically as possible. Automatic machine learning (AutoML) is a new research area that targets the progressive automation of machine learning. One of its success stories is Bayesian hyperparameter optimization which tries to find the best hyperparameter setting for a given machine learning algorithm.

In this talk, I will show how Bayesian optimization can be efficiently used for hyperparameter optimization. I will also present recent advances that speed up the optimization process by exploiting cheap approximations of the objective function, such as the performance when running on a subset of data or the learning curves of iterative machine learning algorithms.

Armand Joulin: Multilingual word alignment

Research scientist, Facebook Artificial Intelligence Research

We consider the problem of aligning continuous word representations, learned in multiple languages, to a common space. It was recently shown that, in the case of two languages, it is possible to learn such a mapping. This talk will present several recent approaches for bilingual alignment that works with and without supervision, as well as an extension to the problem of jointly aligning multiple languages to a common space.

In presence of supervision, we show that this problem can be cast as retrieval problem with a convex formulation, leading to significant improvement over the state of the art. In absence of supervision, we propose an approach based on optimal transport, with theoretical guarantees and competitive empirical performance.

Advances in ML: Theory meets practice, January 27, 2019