Cross-Modal Correspondence Between Speech Sound and Visual Shape Influencing Perceptual Representation of Shape: the Role of Articulation and Pitch
2020
Cross-modal
correspondence is the tendency to systematically map stimulus features across
sensory modalities. The current study explored cross-modal correspondence
between speech sound and shape (Experiment 1), and whether such association can
influence shape representation (Experiment 2). For the purpose of closely
examining the role of the two factors ‒ articulation and pitch ‒ combined in speech acoustics, we
generated two sets of 25 vowel stimuli ‒ pitch-varying and pitch-constant sets.
Both sets were generated by manipulating articulation ‒ frontness and height of the tongue body’s
positions ‒ but differed in terms of
whether pitch varied among the sounds within the same set. In Experiment 1,
participants made a forced choice between a round and a spiky shape to indicate
the shape better associated with each sound. Results showed that shape choice
was modulated according to both articulation and pitch, and we therefore
concluded that both factors play significant roles in sound–shape correspondence.
In Experiment 2, participants reported their subjective experience of shape
accompanied by vowel sounds by adjusting an ambiguous shape in the response
display. We found that sound–shape correspondence exerts an effect on shape
representation by modulating audiovisual interaction, but only in the case of
pitch-varying sounds. Therefore, pitch information within vowel acoustics plays
the leading role in sound–shape correspondence influencing shape
representation. Taken together, our results suggest the importance of teasing
apart the roles of articulation and pitch for understanding sound–shape
correspondence.
Keywords:
- Correction
- Source
- Cite
- Save
- Machine Reading By IdeaReader
98
References
0
Citations
NaN
KQI