Fast and Slow Learning of Recurrent Independent Mechanisms (Machine Learning Paper Explained)

#metarim #deeprl #catastrophicforgetting

Reinforcement Learning is very tricky in environments where the objective shifts over time. This paper explores agents in multi-task environments that are usually subject to catastrophic forgetting. Building on the concept of Recurrent Independent Mechanisms (RIM), the authors propose to separate the learning procedures for the mechanism parameters (fast) and the attention parameters (slow) and achieve superior results and more stability, and even better zero-shot transfer performance.

OUTLINE: 0:00 - Intro & Overview 3:30 - Recombining pieces of knowledge 11:30 - Controllers as recurrent neural networks 14:20 - Recurrent Independent Mechanisms 21:20 - Learning at different time scales 28:40 - Experimental Results & My Criticism 44:20 - Conclusion & Comments

Paper: https://arxiv.org/abs/2105.08710 RIM Paper: https://arxiv.org/abs/1909.10893

Abstract: Decomposing knowledge into interchangeable pieces promises a generalization advantage when there are changes in distribution. A learning agent interacting with its environment is likely to be faced with situations requiring novel combinations of existing pieces of knowledge. We hypothesize that such a decomposition of knowledge is particularly relevant for being able to generalize in a systematic manner to out-of-distribution changes. To study these ideas, we propose a particular training framework in which we assume that the pieces of knowledge an agent needs and its reward function are stationary and can be re-used across tasks. An attention mechanism dynamically selects which modules can be adapted to the current task, and the parameters of the selected modules are allowed to change quickly as the learner is confronted with variations in what it experiences, while the parameters of the attention mechanisms act as stable, slowly changing, meta-parameters. We focus on pieces of knowledge captured by an ensemble of modules sparsely communicating with each other via a bottleneck of attention. We find that meta-learning the modular aspects of the proposed system greatly helps in achieving faster adaptation in a reinforcement learning setup involving navigation in a partially observed grid world with image-level input. We also find that reversing the role of parameters and meta-parameters does not work nearly as well, suggesting a particular role for fast adaptation of the dynamically selected modules.

Authors: Kanika Madan, Nan Rosemary Ke, Anirudh Goyal, Bernhard Schölkopf, Yoshua Bengio

Links: TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick YouTube: https://www.youtube.com/c/yannickilcher Twitter: https://twitter.com/ykilcher Discord: https://discord.gg/4H8xxDF BitChute: https://www.bitchute.com/channel/yannic-kilcher Minds: https://www.mi ... https://www.youtube.com/watch?v=kU-tWy_wr78