Sayaka Kawakami1,2 • Sadao Otsuka3
1National Institute of Occupational Safety and Health Japan, Kawasaki, Japan; 2Faculty of Human Health Sciences, Graduate School of Medicine, Kyoto University, Kyoto, Japan; 3Department of Psychiatry, Graduate School of Medicine, Kyoto University, Kyoto, Japan
Abstract: Regarding the atypical characteristics of cognition and information processing in individuals with autism spectrum disorder (ASD), recent focus has been centered around fundamental processing, such as multisensory integration (MSI). Experimental studies have reported atypical MSI, especially audio-visual integration, in both children and adults with ASD using social (e.g., faces and voices) and nonsocial stimuli (e.g., flashes and beeps). Furthermore, there has been a gradual increase in the understanding of the behavioral (e.g., higher temporal resolution) and neural mechanisms (e.g., impaired phase alignment of neuro-oscillations) underlying atypical MSI in ASD. Previous studies have proposed that prominent deficits in social cognition and interactions (i.e., higher-order functions) are influenced and/or induced by atypicalities in MSI (i.e., lower-order function). Thus, interventions targeting MSI may promote social cognition, likely resulting in better outcomes in adulthood in individuals with ASD. This chapter describes current knowledge regarding multisensory processing in ASD and future perspectives on relevant research and practices. We highlight the value of focusing on MSI for understanding the clinical characteristics of ASD and possible interventions targeting MSI for this population.
Keywords: cascading effect; early intervention; multisensory integration; social cognition; temporal resolution
Author for correspondence: Sayaka Kawakami, National Institute of Occupational Safety and Health Japan, Nagao 6-21-1, Tama-Ku, Kawasaki 214-8585, Japan. Email: k.m.sayaka@gmail.com
Doi: https://doi.org/10.36255/exonpublications.autismspectrumdisorders.2021.multisensoryprocessing
In: Autism Spectrum Disorders. Grabrucker AM (Editor). Exon Publications, Brisbane, Australia. ISBN: 978-0-6450017-8-5; Doi: https://doi.org/10.36255/exonpublications.autismspectrumdisorders.2021
Copyright: The Authors.
License: This open access article is licenced under Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) https://creativecommons.org/licenses/by-nc/4.0/
Regarding the atypical characteristics of cognition and information processing in individuals with autism spectrum disorders (ASD), recently, fundamental processing, such as multisensory integration (MSI), has gained attention, alongside higher-order functions. MSI is the ability of the sensory system to integrate information across different modalities, which offers various behavioral and perceptual benefits. For example, MSI is useful for recognizing the emotions of others by not only facial expressions but also by the tone of voice. Atypicalities in MSI, especially audio-visual integration, have been reported in both children and adults with ASD in studies that used social (e.g., faces and voices) and nonsocial stimuli (e.g., flashes and beeps). Moreover, there has been a gradual increase in our understanding of the behavioral (e.g., higher temporal resolution) and neural mechanisms (e.g., impaired phase alignment of neuro-oscillations) underlying atypical MSI in those with ASD.
Studies focusing on fundamental processing in ASD have proposed that prominent deficits in higher-order functions (e.g., social cognition, communication, and interaction) are influenced and/or induced by alterations in MSI or other lower-order fundamental functions (Figure 1). Where MSI is helpful in social situations, atypicality in MSI impacts social cognition, which is closely associated with adaptive or social functioning (1). Therefore, interventions targeting MSI may promote social cognition, which will result in better outcomes in adulthood in individuals with ASD.
Figure 1. Illustration of the cascading effect. Higher-order functions are influenced by lower-order functions. We propose that interventions targeting atypical multisensory integration in individuals with ASD may promote social cognition and functioning.
In this chapter, we describe current knowledge regarding multisensory processing in ASD and future perspectives on relevant research and practices. We highlight the value of focusing on MSI as a vital element of social cognition and functioning, to understand better the clinical characteristics and possible interventions targeting MSI in those with ASD.
Atypical sensory responses are common clinical symptoms in people with ASD. The Diagnostic and Statistical Manual of Mental Disorders, 5th edition (2) describes these as “hyper- or hypo- reactivity to sensory input or unusual interest in sensory aspects of the environment”. Numerous studies have been conducted on sensory dysfunctions in people with ASD. Reports have indicated that as many as 95% of children with ASD have some degree of sensory dysfunction based on total scores of the Short Sensory Profile (3). Furthermore, in recent decades, experimental studies have revealed that atypicalities in people with ASD are observed not only in a single sensory modality (4) but also in the integration of multiple sensory modalities.
Tables 1 and 2 summarize studies examining MSI in people with ASD, including phenomenon such as the McGurk effect (5). The McGurk effect is a phenomenon whereby a third phoneme is perceived when a video of a phonetic utterance is viewed simultaneously with another phonetic utterance. Although some reports have shown no difference between individuals with ASD and typically developing (TD) people (6–8), many reports have indicated that MSI is impaired in people with ASD (9–12). According to the most recent meta-analysis, individuals with ASD show a weaker McGurk effect than TD people (13), suggesting that people with ASD do not benefit from integration when processing social information. However, because inefficient processing of social information in individuals with ASD likely affects their MSI of social stimuli, a paradigm using simple stimuli without social meaning is more appropriate for purely examining problems with MSI.
TABLE 1 List of studies using the McGurk task
Study | Subjects | Age (mean ± SD, range) | Results of the ASD group |
---|---|---|---|
de Gelder et al. (9) | 17 ASD 17 TD |
10.9 ± 2.3 (6.5–16.3) 8.5 ± 1.3 (6.8–11.1) |
Reduced MSI |
Williams et al. (6) | 15 ASD 15 TD |
8.8 ± 1.4 (5–13) 9.5 ± 1 (5–13) |
No group difference |
Mongillo et al. (10) | 15 ASD 21 TD |
13.7 ± 3.9 (8–19) 13.4 ± 2.8 (11–19) |
Reduced MSI |
Taylor et al. (11) | 24 ASD 30 TD |
12.6 ± 2.4 (7.9–16.4) 11.8 ± 2.5 (8.3–16.4) |
Impaired MSI at a young age but improved with age |
Woynaroski et al. (7) | 18 ASD 18 TD |
12.3 ± 2.6 (8–17) 11.5 ± 1.9 (8–17) |
No group difference |
Bebko et al. (12) | 15 ASD 19 TD |
10.5 ± 2.5 (6.6–14.6) 10.2 ± 2.7 (6–15.6) |
Reduced MSI (in the autism group but not in the Asperger group) |
Stevenson et al. (14) | 32 ASD 32 TD |
11.8 ± 3.2 (6–18) 12.3 ± 2.3 (6–18) |
Reduced MSI |
Keane et al. (8) | 9 ASD 9 TD |
30 ± 9 (19–47) 30 ± 8 (18–49) |
No group difference |
ASD, autism spectrum disorder; MSI, multisensory integration; SD, standard deviation; TD, typically developing
TABLE 2 List of studies using sound-induced flash illusion and temporal order judgment tasks
Study | Subjects | Age (range, mean ± SD) | Task | SOA | Results (Fission illusion) |
---|---|---|---|---|---|
Foss-Feig et al. (15) |
29 ASD 17 TD |
12.6 ± 2.6 (8–17) 12.1 ± 2.2 (8–17) |
SIFI | Operate | Improved MSI (wider TW) for ASD group |
Stevenson et al. (14) |
31 ASD 31 TD |
12.1 ± 3.1 (6–18) 11.9 ± 2.9 (6–18) |
SIFI | Did not operate (varied the number of flashes) | Reduced MSI for ASD group |
Van der Smagt et al. (16) |
15 ASD 15 TD |
20.5 ± 3.2 20.7 ± 2.6 |
SIFI | Did not operate (varied the number of flashes) | No group differences |
Bao et al. (17) |
20 ASD 20 TD |
18.7 ± 4.7 (13–29) 18 ± 9.5 (13–28) |
SIFI | Did not operate | No group difference (but larger susceptibility to the fusion illusion) |
Keane et al. (8) |
6 ASD 6 TD |
N/A | SIFI | Operate | No group difference |
Kawakami et al. (18) |
84 TD | 21.1 ± 2.4 | SIFI | Operate | Reduced MSI (lower sensitivity to SIFI was related to higher autistic traits) |
Kawakami et al. (19) |
15 ASD 18 TD |
28.13 ± 7.16 29.00 ± 10.39 |
SIFI | Operate | Reduced MSI (lower sensitivity to SIFI in ASD group) |
Kwakye et al. (20) |
35 ASD 27 TD |
12.2 ± 2.7 (8–17) 11.7 ± 2.4 (8–17) |
TOJ | Operate | Wider TW for ASD |
Kawakami et al. (18) |
84 TD | 21.1 ± 2.4 | TOJ | Operate | Narrower TW was related to higher autistic traits |
Kawakami et al. (19) |
15 ASD 18 TD |
28.13 ± 7.16 29.00 ± 10.39 |
TOJ | Operate | No group difference |
Casassus et al. (21) |
18 ASD 18 TD |
31 ± 8.4 31 ± 8.7 |
TOJ | Operate | No group difference |
ASD, autism spectrum disorder; MSI, multisensory integration; SD, standard deviation; TD, typically developing; TW, temporal window; SOA, stimulus onset asynchrony
A paradigm that can be used to investigate MSI that uses simple stimuli without social information is the sound-induced flash illusion (SIFI) task (22). In this paradigm, one or two flashes with or without beeps are presented, and the participant is asked to count the number of flashes while ignoring the beeps. In the illusory condition, one flash is accompanied by two beeps. If an individual is able to integrate visual and auditory information effectively, two flashes are perceived. Other conditions include one (or two) beep(s) only trial, one (or two) flash(es) only trial, and two sets of flash and beep trials, which are presented in random order.
Numerous studies have investigated MSI in people with ASD using the SIFI task. However, the results have been inconsistent (Table 2). Some have reported that people with ASD and TD show equal susceptibility to the SIFI task (8, 16, 17). In contrast, others have shown that individuals with ASD have reduced (14, 19) or enhanced (15) susceptibility to the SIFI task.
Such inconsistencies may be attributed to a developmental delay rather than a persistent difference (23). Although children with ASD show reduced susceptibility to this illusion relative to TD controls (14), adults with ASD appear to be equally susceptible to the illusion as controls (8, 16, 17). Moreover, when age is controlled for, both adolescents and adults with ASD show lower sensitivity to the SIFI task (19, 18). These findings suggest that the development of MSI is delayed in individuals with ASD.
Recent studies have suggested that more basic functions may be related to MSI, namely, temporal resolution. Information that needs to be integrated is often simultaneous or very close in time, such as when an object falls (visual information), and a cracking sound is heard (auditory information) simultaneously (Figure 2). The period in which plural inputs are perceived as simultaneous is called a temporal window (TW). Temporal resolution changes with development alongside MSI ability. The developmental change of a TW occurs during and after adolescence (21). Stevenson et al. (24) showed that temporal acuity for audio-visual information improves until around middle age and declines with age. If basic information processing abilities are altered in people with ASD, they may encounter difficulties with higher-order functions (i.e., MSI and social cognition using MSI (Figure 3)).
Figure 2. Illustration of a cup falling and breaking. Information that needs to be integrated is very close in time. If an excessively short time lag is recognized as simultaneous, the occurrence of multisensory integration becomes low. Moreover, the time lag between inputs that need to be integrated is typically longer for social information than physics-related information.
Figure 3. A hypothetical model. Illustration of our hypothetical model on the impact of developmental delays in sensory processing on the later development of higher-order functions in people with ASD. From a developmental perspective, multisensory integration is thought to partly underlie social cognition, which in turn underlies social interactions and functioning. Thus, if treatment can promote the development of multisensory integration in individuals with ASD, it may improve social cognition, core symptoms, and social functioning. ASD, autism spectrum disorder.
Alterations in basic information processing may be related to difficulties in higher-order functions. If there are differences in basic information processing ability in individuals with ASD, they may encounter problems with higher-order functions (i.e., MSI and social cognition using MSI). However, this also means that interventions targeting atypical MSI in people with ASD may promote social cognition and social functioning.
Several reports have shown that temporal resolution is related to MSI ability (18, 25). Stevenson et al. (25) conducted an experiment in the general population and found that higher temporal resolution is related to lower occurrences of MSI. Similarly, we found that the stronger the ASD characteristics in the general population, the higher the temporal resolution and the lower the likelihood that MSI will occur (18). We also conducted the same experiment in individuals with high-functioning ASD and reported a similar trend (19), which suggested that MSI is affected by more fundamental information processing characteristics. Therefore, interventions targeting lower-order functions may also improve higher-order functions.
Crucially, the narrowness of TW may make MSI difficult, especially for social information. Our study (18) found that both a narrow TW and difficulties in MSI are related to problems with social skills, as measured by a subscale of the autism-spectrum quotient (AQ), and MSI mediates the relationship between TW and social skills. This suggests that MSI plays an essential role in social interactions at specific points in time (e.g., recognizing others’ facial expressions at a particular point in time) and in series of events that flow (e.g., understanding others’ emotions according to the flow of the scene). Furthermore, individuals who have a narrow TW may have difficulty with MSI, especially in social situations, because the time lag between inputs that need to be integrated is typically longer for social information than physics-related information.
For MSI, neuronal rhythms (i.e., neuro-oscillatory functions, which are rhythmic and/or repetitive electrical activity generated spontaneously by neural tissue in response to stimuli) play an important role (26–28). Even subtle disturbances in the coordination of inter-regional phase relationships may lead to dysfunctional information processing. Indeed, it has been suggested that there may be abnormalities in neuro-oscillatory functions in people with ASD (29). For example, gamma-band perturbations in autism are commonly observed. Balz et al. (30) reported that proneness to the SIFI task was correlated with gamma-aminobutyric acid levels, which have been reported to be altered in individuals with ASD (31). If the phase of neuronal oscillations plays a critical role in sensory integration, even subtle differences in this mechanism may lead to malfunctions in information processing.
As a neural mechanism associated with MSI deficits in ASD, Beker et al. (23) proposed that regional neural connections are weaker in those with ASD, resulting in a non-synchronous intersensory activity. It is possible that advanced processing stages, including MSI, are more affected than basic sensory information processing. These potentially cascading effects are likely to affect the development of various cognitive abilities that are commonly impaired in people with ASD. For example, as a result of atypical multisensory integration resulting from non-synchronous intersensory activity, the development of the abilities to recognize others’ emotions or to read others’ intentions by integrating facial (visual) and vocal (auditory) information could be delayed.
Atypical MSI in people with ASD significantly impacts cognition and behavior during social situations (Figure 3). For example, in a noisy environment, it is easier to understand what an individual is saying by observing the movement of their mouth (32). Similarly, it is not only visual information, such as facial expressions, but also auditory information that is considered for judging an individual’s emotional state (33, 34). This aids in the recognition of complex emotional states in others, depending on the specific context (e.g., a forced smile by a sorrowful individual). Another study showed that problems in sensory processing are associated with stress in general laborers (35), where ease of excitation and low sensory thresholds are related to job demands and emotional exhaustion. Therefore, MSI is vital for effective communication in social situations, and it is likely that MSI ability is related to the core symptoms of ASD. From a developmental perspective, atypical characteristics of MSI may impact social learning through social interactions during development, and social-cognitive processes that involve MSI are thought to be acquired during development. Conversely, approaches to basic functions, such as MSI, from an early stage may positively impact the subsequent development of higher functions.
MSI is important in social situations. One can hypothesize that atypicalities in MSI and related basic functions in people with ASD are due to delayed development rather than innate differences. In addition, we propose a cascade model hypothesizing that delayed MSI in people with ASD affects higher-order functions, such as social cognition, interaction, and functioning (Figure 3). If the development of MSI in people with ASD is delayed rather than altered, interventions that target and promote the development of MSI would be promising. Furthermore, such interventions may improve and promote the development of higher-order social functions that are considered partially dependent on MSI.
As for the general adults, there is a report that temporal discrimination of multisensory stimuli, which is shown to associate with MSI performance in people with ASD (14), can be enhanced with training. This study employed a perceptual learning paradigm, in which participants judged whether the occurrence of a visual stimulus and an auditory stimulus were “simultaneous” or not and were given feedback as to the correctness, and reported the durable effect of narrowing the size of TW (36). In addition, it has been reported that multisensory training for individuals with some kind of visual defects can offer benefits provided by MSI in visual processing (37). For example, systematic audio-visual stimulation for hemianopic patients, in which participants explore the blind hemifield by shifting their gaze toward visual stimulus that follows auditory stimulus, was shown to improve not only visual perception during training but also visual field exploration and daily life activities (38). Those results suggest that multisensory interventions can produce the intended effects on sensory functions, which may in turn facilitate higher-order daily and social functioning.
An example of a multisensory intervention for children with neurodevelopmental disorders is infant massage (IM), a standardized practice developed by the International Association of Infant Massage that uses various ways of touch, movement, interaction, and communication to foster intensive and affective multisensory stimulation. During a massage, infants receive information from different sensory modalities (e.g., voice, touch, kinesthetic manipulation, and facial expressions) from the caregiver. In a study examining the effects of IM on infants with neurodevelopmental disorders, Purpura et al. (39) reported that multisensory intervention, including IM, positively affects visual system development (e.g., visual acuity and stereopsis maturation) in infants with Down Syndrome.
Another intervention that targets multisensory integration in children with neurodevelopmental disorders is Ayres Sensory Integration, which focuses on sensory functions. Although this intervention incorporates elements that are thought to challenge scientific rigor, two randomized controlled trials (RCT) studies have been conducted (40), which showed that it is effective in children aged 4–12 years with ASD (41, 42).
The interventions described above show promise for promoting the development of MSI and other basic functions in children with ASD. Furthermore, the development of these basic functions may underlie those of social cognition that predict outcomes in adulthood (1). In adults with ASD, it would also be valuable to develop MSI interventions that target social cognition, such as training individuals to recognize emotions from faces, voices, and other cues and integrating these skills in daily life (e.g., Social Cognitive Skills Training; (43, 44)). Future studies are needed to examine the effects of such interventions in people with ASD using rigorous scientific methodologies.
Individuals with ASD have delayed development with impairment of MSI. MSI and other basic functions may underlie the social cognition of those with ASD. Therefore, interventions that target MSI, especially in early childhood, may promote the development of social cognition, which will improve social interactions and social functioning in this population. Interventions aimed at reducing the core symptoms of ASD, such as problems with interpersonal interactions, are currently being implemented; however, few treatments are available in clinical settings. Mainstream interventions are aimed at acquiring specific compensatory skills, treating secondary disorders, and modifying environments. Future studies are needed to develop interventions targeting MSI and other basic functions in people with ASD and examine their efficacy using rigorous scientific methodologies. Furthermore, it will be imperative to test the long-term impact on social cognition and interpersonal interactions in people with ASD.
Acknowledgment: This work was supported by JSPS KAKENHI Grant Number JP 18K13345. We thank Sarina Iwabuchi, PhD, from Edanz (https://jp.edanz.com/ac) for editing a draft of this manuscript.
Conflict of interest: The authors declare no potential conflict of interest with respect to research, authorship, and/or publication of this chapter.
Copyright and permission statement: The authors confirm that the materials included in this chapter do not violate copyright laws. Where relevant, appropriate permissions have been obtained from the original copyright holder(s), and all original sources have been appropriately acknowledged or referenced.