Gradient Origin Networks (Paper Explained w/ Live Coding)

Neural networks for implicit representations, such as SIRENs, have been very successful at modeling natural signals. However, in the classical approach, each data point requires its own neural network to be fit. This paper extends implicit representations to an entire dataset by introducing latent vectors of data points to SIRENs. Interestingly, the paper shows that such latent vectors can be obtained without the need for an explicit encoder, by simply looking at the negative gradient of the zero-vector through the representation function.

OUTLINE: 0:00 - Intro & Overview 2:10 - Implicit Generative Models 5:30 - Implicitly Represent a Dataset 11:00 - Gradient Origin Networks 23:55 - Relation to Gradient Descent 28:05 - Messing with their Code 37:40 - Implicit Encoders 38:50 - Using GONs as classifiers 40:55 - Experiments & Conclusion

Paper: https://arxiv.org/abs/2007.02798 Code: https://github.com/cwkx/GON Project Page: https://cwkx.github.io/data/GON/

My Video on SIREN: https://youtu.be/Q5g3p9Zwjrk

Abstract: This paper proposes a new type of implicit generative model that is able to quickly learn a latent representation without an explicit encoder. This is achieved with an implicit neural network that takes as inputs points in the coordinate space alongside a latent vector initialised with zeros. The gradients of the data fitting loss with respect to this zero vector are jointly optimised to act as latent points that capture the data manifold. The results show similar characteristics to autoencoders, but with fewer parameters and the advantages of implicit representation networks.

Authors: Sam Bond-Taylor, Chris G. Willcocks

Links: YouTube: https://www.youtube.com/c/yannickilcher Twitter: https://twitter.com/ykilcher Discord: https://discord.gg/4H8xxDF BitChute: https://www.bitchute.com/channel/yannic-kilcher Minds: https://www.minds.com/ykilcher Parler: https://parler.com/profile/YannicKilcher ... https://www.youtube.com/watch?v=v-ZxzTSpmk4