This project created theories of how the brain could represent the world and use that representation for behaving. It focused on two key principles of representation: uncertainty and nonlinearity. For the first topic, uncertainty, we begin with the premise that animals receive lots of sensory data, but not all of it is equally reliable. For a familiar example, it’s usually harder to hear where a sound comes from than to see the thing that makes the sound. For the brain to adjust how much to trust different sensory inputs compared to its mental predictions, it must somehow represent the reliabilities of all of these factors. We developed and contrasted theories of how brain activity might encode these reliabilities, which we view through the lens of probability and statistics. This allowed us to address a major challenge, namely how those probabilities could account for causal connections between variables. For this we described brain implementations of a mathematical framework called probabilistic graphical models, and showed how this could avoid a scaling catastrophe that previous work claimed would plague some theories about brain computations.

            The second topic, nonlinearity, is a core ingredient in modern algorithms for machine intelligence. Nonlinearity refers to “folding” signals so that related information can be processed similarly. This is like folding and cutting a paper snowflake, which can make complicated shapes out of simple operations. In the brain and in machines, this approach can be used to find complicated data patterns using simple hardware. We developed methods to understand what nonlinearities the brain actually uses, by analyzing how neural data relates to both the brain’s sensory inputs and the behavioral outputs. We used this approach to analyze neural recordings from monkey brains, and showed that some animals actually use most of the information their brain has about some tasks.

            These two topics are also closely connected, because representing and using uncertainty generally requires nonlinear computation. Within this project, we’ve connected these ideas to understand abstract theories and concrete analyses of experiments. On the abstract side, we developed theories about when the brain should try to suppress predicted inputs; theories about how to bridge between two prominent, but quite distinct, machine learning algorithms to get the best of both; and theories about how we incorporate more of the most basic nonlinearities of biological neurons into richer units within artificial neural networks. On the concrete side, we analyzed neural data to discover: how uncertainties affect monkey navigation in virtual reality; which nonlinear combinations of visual inputs best activate neurons in the visual cortex of mice; and how brain states shift in human brains while people speak.

            We also used resources from this grant to develop and disseminate educational material. The largest impact has been my efforts to service on the Executive Committee that led the Neuromatch Academy in 2020. This was a massive online summer school in computational neuroscience that taught nearly 7000 students from over 100 countries, with live instruction in 13 languages, and prerecorded content captioned in three languages. This intense effort was organized rapidly over three months to partially repair the educational disruption caused by the COVID-19 pandemic, as all summer schools were canceled and those formative experiences were lost. I led multiple teams of people by chairing the Projects committee and through contributions to the Curriculum, Professional Development, Diversity, and Mentoring Committees. All of the course materials will remain free online for anyone. I continued this engagement in 2021–2 to improve the course materials and interactivity. Overall we established a new standard for delivering high quality remote interactive content with a focus on inclusivity, making lemonade out of pandemic lemons.

Last Modified: 10/14/2022
Modified by: Xaq Pitkow