Santa Fe Institute Collaboration Platform

COMPLEX TIME: Adaptation, Aging, & Arrow of Time

Get Involved!
Contact: Caitlin Lorraine McShea, Program Manager,

An Optimization-Based Approach to Understanding Sensory Systems

From Complex Time
General Reference
author-supplied keywords
Daniel Yamins
An Optimization-Based Approach to Understanding Sensory Systems
Cognitive Neuroscience
W. W. Norton & Company


Recent results have shown that deep neural networks (DNN) may have significant potential to serve as quantitatively precise models of sensory cortex neural populations. However, the implications these results have for our conceptual understanding of neural mechanisms are subtle. This is because many modern DNN brain models are best understood as the products of task-constrained optimization processes, unlike the intuitively simpler hand-crafted models from earlier approaches. In this chapter, we illustrate these issues by first discussing the nature of information processing in the primate ventral visual pathway, and review results comparing the response properties of units in goal-optimized DNN models to neural responses found throughout the ventral pathway. We then show how DNN visual system models are just one instance of a more general optimization framework whose logic may be applicable to understanding the underlying constraints that shape neural mechanisms throughout the brain.

An important part of a scientist's job is to answer "why" questions. For cognitive neuroscientists, a core objective is to uncover the underlying reasons why the structures of the human brain are as they are. Since brains are biological systems, answering such questions is ultimately a matter of identifying the evolutionary and developmental constraints that shape brain structure and function. Such constraints are in part architectural: what large-scale brain structures are put in place genetically to help a brain help its host organism better meet evolutionary challenges? In light of the centrality of behavior in understanding the brain, an ethological investigation is also indicated: what behavioral goals most strongly constrain a given neural system? And since many complex behaviors in higher organisms are not entirely genetically determined and must instead be partly derived through experience of the world, a core question of learning is also involved: how do learning rules that absorb experiential data constrain what brains look like?

The interactions between architectural structure, behavioral goals, and learning rules suggest a quantitative optimization framework as one route toward answering these "why" questions. Put simply, this means: postulating one or several goal behavior(s) as driving the evolution and/or development of a neural system of interest; finding architecturally plausible computational models that (attempt to) optimize for the behavior; and then quantitatively comparing the internal structures arrived at in the optimized models to measurements from large-scale neuroscience experiments. To the extent that there is a match between optimized models and the real data that is very substantially better than that found for various controls (e.g. models designed by hand or optimized for other tasks), this is evidence that something important has been understood about the underlying constraints that shape the brain system under investigation. Though it might sound challenging to put this approach into practice, recent successes suggest we might add to our list of maxims the observation that nothing in computational cognitive neuroscience makes sense except in light of optimization.


Citation count From Scopus. Refreshed every 5 days.
Page views


Related files

Add a file