Physics Colloquium: Tankut Can, "Dynamics of Gated Recurrent Neural Networks "

The City College of the City University of New York
160 Convent Avenue
New York, NY 10031

Physics Department
Marshak Science Building, Room 419
Phone: 212-650-6832
fax: 212-650-6940
email: physdept@ccny.cuny.edu
-----------------------------------------------------------------------------------------

Wednesday, September 16, 2020 from 04:00 PM to 05:00 PM

Where                         Zoom

Contact Name                         Sriram Ganeshan

Contact Email                         sganeshan@ccny.cuny.edu

 

Physics Colloquium

 

Dynamics of Gated Recurrent Neural Networks 

Tankut Can
Postdoctoral Fellow
Institute for Theoretical Sciences
Graduate Center, CUNY
New York, NY

Abstract

Recurrent neural networks (RNNs) have long captured the attention and imagination of the scientific community. Initially  studied as a model for the brain, they have now established a firm foothold in the modern practice of machine learning. RNNs can now achieve state-of-the-art performance on sequential learning tasks like speech recognition, translation, and text prediction, in large part due to a key feature: gating, a multiplicative interaction which controls the flow of information. Despite this progress, we still do not really know how gating shapes the dynamics of RNNs, which presents a hurdle to building even better learning systems.

 

In this talk, I will show how physics approaches can shed light on why gated RNNs work, as well as how they can be improved. Furthermore, I will argue that gated RNNs are interesting in their own right as models of complex interacting systems, and give rise to several novel phenomena, including robust line attractors and a first-order chaotic transition. Finally, I hope to convey a sense of the excitement in this growing field at the intersection of physics and machine learning.

--------------------------

 

The goal of ITS is to provide a home for research and teaching, emphasizing theoretical approaches to the natural sciences.  Participants are interested in a wide range of phenomena, from the elementary building blocks of matter to the dynamics of social systems, from the collective behavior of electrons in solids to the collective behavior of neurons in the brain, and more.  What ties these disparate problems together is the search for a compact and compelling mathematical description of the world around us.