Embedded Learning and Sensing Systems Group�ASAI networking event @ICML’24�
Olga Saukh, saukh@tugraz.at �
Embedded Learning and Sensing Systems Group, teams.iti.tugraz.at/elss
Institute of Technical Informatics
Graz University of Technology
AI & Machine Learning
Complexity Science Hub
Embedded Learning & Sensing Systems
Group: Embedded Learning and Sensing Systems (ELSS)
Olga Saukh�(group leader)
Francesco Corti�(PhD student)
Nam Cao�(PhD student)
Katarina Petrovic�(PhD student)
Dong Wang�(PhD student)
Franz Papst�(PhD student)
Vision: Towards Trusted Edge Intelligence
Less is More. Small can be Mighty.
Adaptation / reconfiguration
Trustworthy AI
Reality “check”
Optimization for DL
Selected Topic: Linear mode connectivity of independently� trained deep models in the weight space
Permutation Conjecture�Taking permutations into account, there is likely no barrier �on the linear interpolation between SGD solutions
R. Entezari et al., The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks, ICLR 2022
K. Jordan et al., REPAIR: Renormalizing Permuted Activations for Interpolation Repair, ICLR 2023
Permutation invariance
C
A
B
A
B
C
≡
Models are invariant to the permutation of neurons as long as the incoming and outgoing weights are permuted too
Selected Topic: Reconfigurable deep learning for edge intelligence
Due to severe resource constraints edge devices often lack support for on-device learning / backprob
Can a model be reconfigured instead of adapted?
-10°
+10°
D. Wang et al., Subspace-Configurable Networks, CoLLAs 2024 �(Oral presentation)
We train a low-dimensional configuration subspace (𝜷-space) used to construct inference network as a linear combination of base model weights
Subspace-configurable �networks make invariances explicit
Join us to discuss recent advances
… and help to spread the word
public