Noether Networks: Meta-Learning Useful Conserved Quantities (w/ the authors)
Yannic Kilcher
#deeplearning #noether #symmetries
This video includes an interview with first author Ferran Alet! Encoding inductive biases has been a long established methods to provide deep networks with the ability to learn from less data. Especially useful are encodings of symmetry properties of the data, such as the convolution's translation invariance. But such symmetries are often hard to program explicitly, and can only be encoded exactly when done in a direct fashion. Noether Networks use Noether's theorem connecting symmetries to conserved quantities and are able to dynamically and approximately enforce symmetry properties upon deep neural networks.
OUTLINE: 0:00 - Intro & Overview 18:10 - Interview Start 21:20 - Symmetry priors vs conserved quantities 23:25 - Example: Pendulum 27:45 - Noether Network Model Overview 35:35 - Optimizing the Noether Loss 41:00 - Is the computation graph stable? 46:30 - Increasing the inference time computation 48:45 - Why dynamically modify the model? 55:30 - Experimental Results & Discussion
Paper: https://arxiv.org/abs/2112.03321 Website: https://dylandoblar.github.io/noether-networks/ Code: https://github.com/dylandoblar/noether-networks
Abstract: Progress in machine learning (ML) stems from a combination of data availability, computational resources, and an appropriate encoding of inductive biases. Useful biases often exploit symmetries in the prediction problem, such as convolutional networks relying on translation equivariance. Automatically discovering these useful symmetries holds the potential to greatly improve the performance of ML systems, but still remains a challenge. In this work, we focus on sequential prediction problems and take inspiration from Noether's theorem to reduce the problem of finding inductive biases to meta-learning useful conserved quantities. We propose Noether Networks: a new type of architecture where a meta-learned conservation loss is optimized inside the prediction function. We show, theoretically and experimentally, that Noether Networks improve prediction quality, providing a general framework for discovering inductive biases in sequential problems.
Authors: Ferran Alet, Dylan Doblar, Allan Zhou, Joshua Tenenbaum, Kenji Kawaguchi, Chelsea Finn
Links: TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick YouTube: https://www.youtube.com/c/yannickilcher Twitter: https://twitter.com/ykilcher Discord: https://discord.gg/4H8xxDF BitChute: https://www.bitchute.com/channel/yannic-kilcher LinkedIn: https://www.linkedin.com/in/ykilcher BiliBili: https://space.bilibili.com/2017636191
If you want to support me, the best thing to do is to share out the content :)
If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
414301061 Bytes