Barry E. Stein, PhD

Multisensory Research Group


Development of IntegrationIntegration in the CatNeural Network



The anatomical, physiological, behavioral, and computational bases for the development and expression of multisensory integration.

Decoding and interpreting incoming sensory information are among the brain's most important tasks. These are ongoing processes that make it possible for us not only to know the world in which we live, but to plan and initiate behaviors that are appropriate for a particular circumstance. Because survival depends on the speed and accuracy of such processes, it is not surprising to find that encoding, decoding, and evaluating sensory information have been powerful driving forces in evolution. Consequently, extant organisms have an impressive array of specialized sensory systems.

Having multiple sensory systems provides significant benefits; it allows an organism to monitor simultaneously a host of environmental cues, and also provides a means of substituting one sensory system for another when necessary (e.g., hearing and/or touch can substitute for vision in the dark). The ability to monitor and process multiple sensory cues in “parallel” not only increases the likelihood that a given stimulus will be detected, but, because the information carried along each sensory channel reflects a different feature of that stimulus, it increases the chances that the stimulus will be properly identified. Stimuli that may be difficult to distinguish by means of a single sensory modality (e.g., how they look) can become quite distinct via information from another modality (how they sound or feel).

Of particular benefit is the brain’s ability to use the information carried by these different sensory channels synergistically. Thus, different sensory inputs can enhance one another’s effectiveness, increasing the likelihood that an event will be detected, properly identified, and that an appropriate response is initiated as fast as is possible.

Our research goal is to determine how information from different senses is pooled in making decisions, and how the brain develops this remarkable capacity. We use two interrelated neural models to do this: one in the midbrain superior colliculus (SC) and one in association cortex. The midbrain SC has proved to be a most effective model, but its principal features have been shown also to be operative in cortex. Its advantage is not only its host of sensory inputs (visual, auditory, somatosensory), but its many multisensory neurons, and its distinct behavioral function: faciliatating the detection, localization and orientation to external events.

Current Projects

Contrasts In How The Brain Integrates Within-Modal and Cross-Modal Information:Despite the substantial literature describing the impact of multisensory integration in multiple brain areas, and the importance of this process for perception and behavior, a fundamental question has been largely ignored: does the fusion of information from different sensory source yield a product that is different from the fusion of information from the same sensory source?  We are investigating this question at physiological and behavioral levels.

Anatomical Bases of Multisensory Integration: Physiological and behavioral studies have revealed that SC multisensory integration depends on the functional integrity of converging projections that descend from different regions of association cortex. We are currently investigating the nature of these projections and the development of this cortico-SC circuit using modern neuroanatomical tracing techniques and electron microscopy.

Computational Bases of Multisensory Integration: Recent evidence suggests that multisensory integration at the physiological level involves both linear and nonlinear computations.  We are currently developing and testing a neural network model that seeks to explain a multitude of empirical findings within a single framework.

Effects of Multisensory Integration on Response Timing and Information: The magnitude of the physiological impact of integration at the single neuron level has traditionally been measured as a change (usually expressed as a % change) in the total number of stimulus-elicited impulses.  We have recently found that concordant cross-modal stimuli not only produce more robust responses, but significantly speeds these responses.  In addition, we have found that integration produces substantial enhancements in information transmission; conveying not only more information, but information at a faster rate.

Similarities and Differences in Integration Across Species: The traditional animal model of SC multisensory integraiton is the cat, and the anatomical circuit underlying integration has been well-described in this model.  We have recently begun evaluating the generality of this anatomical circuit across mammalian species. 

Experiential and Anatomical Dependencies of the Development of Integration: Our recent evidence suggests that capacity to engage in multisensory integration is not innate capabiility, but rather one that the brain develops after birth in an experience-dependent fashion.  We are currently investigating the constraints guiding its appearance and maturation; in particular, the anatomical circuits involved, the essential properties of the cues that are critical to link them to one another, and whether the strategies of integration change during maturation.

Multisensory Integration in Motion Perception: Natural environments are dynamic, and salient targets are frequently in motion.  Animals interact appropriately with these targets by estimating their velocity and direction of movement.  We are current investigating how the brain combines information from different senses to affect and improve these decisions.


Current Members

  • Dr. Barry E. Stein – Professor and Chair. Interests: Multisensory integration, its circuitry and critical experiential and developmental antecedents.
  • Dr. Thomas Perrault, Jr. – Assistant Professor. Cortical-subcortical interactions in multisensory integration.
  • Dr. Benjamin Rowland – Assistant Professor. Interests: Computational bases of multisensory integration, temporal dynamics of integration.
  • Dr. J. "Bill" Vaughan – Assistant Professor. Interests: Developmental antecedents of multisensory integration, physiological correlates of behavior.

Current Graduate/Professional Students

  • Caterina Bertini (Graduate)
  • Ryan Miller (Graduate)
  • Liping Yu (Graduate)
  • Hermes Hernandez (Medical)


Educational Resources

Selected Books

Stein, B.E. & Meredith, M.A. (1993).  The merging of the senses. MIT Press: Cambridge, MA.
Calvert, G.A., Spence, C., & Stein, B.E. (2004). The handbook of multisensory processes. MIT Press: Cambridge, MA.

Selected Recent Journal Articles: Updated 07/07/2009

Jiang, H., Stein, B.E, and McHaffie, J.G. Opposing basal ganglia processes shape midbrain visuomotor activity bilaterally. Nature 423: 982-986, 2003.

Wallace, M.T., Ramachandran, R. and Stein, B.E. A new view of sensory cortical parcellation. Proc. Natl. Acad. Sci 101(7): 2167-2172, 2004.

Wallace, M.T., Perrault, T.P., Hairston, W.D. and Stein, B.E. Visual experience is necessary for the development of multisensory integration. J. Neurosci. 24: 9580-9584, 2004.

Stein, B.E.  The development of a dialogue between cortex and midbrain to integrate multisensory information.  Exp. Brain Res. 166: 305-315, 2005.

Stanford, T.R., Quessy, S., and Stein, B.E.  Evaluating the operations underlying multisensory integration in cat superior colliculus.  J. Neurosci.  25(28): 6499-6508, 2005.

Alvarado, J.C., Vaughan, J.W., Stanford, T.R., and Stein, B.E.  Multisensory versus unisensory integration: contrasting modes in the superior colliculus.  J. Neurophysiol. 97: 3193-3205, 2007.

Rowland, B.A., Quessy, S., Stanford, T.R., and Stein, B.E.  Multisensory integration shortens physiological response latencies.  J. Neurosci. 27: 5879-5884, 2007.

Jiang, W., Jiang, H., Rowland, B.A., and Stein, B.E.  Multisensory orientation behavior is disrupted by neonatal cortical ablation.  J. Neurophysiol. 97(1): 557-562, 2007.

Rowland, B.A., Stanford, T.R., and Stein, B.E.  A model of the neural mechanisms underlying multisensory integration in the superior colliculus.  Perception [Special Issue on Multisensory Integration] 36: 1431-1443, 2007.

Alvarado, J.C., Stanford, T.R., Vaughan, J.W., and Stein, B.E.  Cortex mediates multisensory but not unisensory integration in superior colliculus.  J. Neurosci. 27(47): 12775-12786, 2007.

Rowland, B.A. and Stein, B.E.  Multisensory integration produces an initial response enhancement.  Frontiers in Integrative Neuroscience 1(4): 1-8, 2007.

Stein, B.E. and Stanford, T.R.  Multisensory integration: current issues from the perspective of the single neuron.  Nature Rev. Neurosci. 9(4): 255-266, 2008.

Gingras, G., Rowland, B.A., and Stein, B.E. The differing impact of multisensory and unisensory integration on behavior.  J. Neurosci.  29(15): 4897-4902, 2009.

Jiang, H., Stein, B.E., and McHaffie, J.G.  Cortical lesion-induced visual hemineglect is prevented by NMDA antagonist pretreatment.  J. Neurosci. 29(21): 6917-6925, 2009.

Alvarado, J.C., Stanford, T.R., Rowland, B.A., Vaughan, J.W., and Stein, B.E.  Multisensory integration in the superior collicullus requires synergy among corticocollicular inputs.  J. Neurosci. 29(20): 6580-6592, 2009.

Stein, B.E., Stanford, T.R., Ramachandran, R., Perrault Jr., T.J., and Rowland, B.A. Challenges in quantifying multisensory integration: Alternative criteria, models, and inverse effectiveness.  Exp Brain Res. [Epub ahead of print], 2009.

Stein, B.E., Stanford, T.R., and Rowland, B.A.  The neural basis of multisensory integration in the midbrain:  Its organization and maturation.  Hearing Research [Epub ahead of print], 2009.

Stein, B.E., Rowland, B., and Stanford, T.R.  Postnatal experiences influence how the brain integrates information from different senses to produce adaptive behavior.  Frontiers in Integrative Neuroscience [Invited Article] (In Press), 2009.

View the full list of journal articles.

Quick Reference

Neurobiology & Anatomy

Phone 336-716-4368

Dr. Barry E. Stein

Dr. Barry E. Stein, Chairman
Department of Neurobiology and Anatomy
Wake Forest School of Medicine
Medical Center Boulevard
Winston-Salem, NC  27157-1010
Find A Doctor Ways to Give
Last Updated: 04-02-2014
USNWR 2013-2014Magnet Hospital RecognitionConsumer Choice2014 Best DoctorsJoint Commission Report

Disclaimer: The information on this website is for general informational purposes only and SHOULD NOT be relied upon as a substitute for sound professional medical advice, evaluation or care from your physician or other qualified health care provider.