Seminar Series

See what you hear – The neural basis of audiovisual integration

To interact effectively with our environment, the human brain integrates information from multiple senses. While multisensory integration was traditionally assumed to be deferred until later processing stages in higher order association cortices, more recent studies have revealed multisensory integration even in putatively unisensory cortical areas. Given this multitude of multisensory integration sites, characterizing their functional similarities and differences is of critical importance. Combining functional imaging, effective connectivity analyses and psychophysics in humans, our studies highlight three main aspects: First, the locus of multisensory integration depends on the type of information being integrated and the specific relationship between the auditory and visual signals. In particular, our data dissociated automatic integration of spatio-temporal features in primary sensory cortices, perceptual integration in association cortices and multisensory decision processes in left prefrontal cortex. Second, in terms of functional brain architectures, effective connectivity analyses suggested that audiovisual interactions in low level sensory areas are mediated by multiple mechanisms including feedforward thalamocortical, direct connections between sensory areas and top down influences from higher order association areas. Third, the profile of audiovisual interactions depends on the reliability and informativeness of the unisensory signals. Predominantly subadditive interactions may turn into superadditive interactions for near threshold inputs. Crucially, these operational profiles of audiovisual integration are functionally relevant and parallel behavioral indices of multisensory enhancement gained from psychophysics.