Special Sessions

Sound Field Modeling and Speech Perception Based on Microphone Arrays in Dynamic Scenarios
Release Time:2026/7/22 12:08:11
Official Website: www.ncmmsc.org.cn

With the rapid development of artificial intelligence, large models, and embodied intelligence technologies, speech interaction is gradually evolving from static, single-person, near-field environments to open, complex, and dynamic real-world scenarios. Emerging applications such as human-computer interaction, smart homes, intelligent cockpits, intelligent robots, and mixed reality impose higher demands on speech processing algorithms: not only the ability to "hear and hear clearly," but also accurate perception of dynamic acoustic environments, distinguishing and understanding spatial relationships among multiple sound sources, and achieving stable and robust speech perception capabilities in real-world scenarios.

As an important carrier for spatial acoustic information acquisition, microphone arrays can provide rich spatiotemporal information and serve as a fundamental basis for dynamic sound field modeling, spatial audio analysis, and robust speech perception. In recent years, traditional array signal processing has been continuously integrated with deep learning, large models, and generative artificial intelligence, achieving significant progress in high-precision sound field reconstruction, dynamic sound source perception, speech enhancement, and sound source separation. Meanwhile, with the development of self-supervised pre-training methods and multimodal foundation models, how to organically combine array observation information, spatial acoustic theory, and data-driven models to construct a new generation of spatial speech intelligence systems with environmental understanding capabilities has become an important research hotspot in the international speech and acoustics community.

This special session focuses on sound field modeling and speech perception based on microphone arrays in dynamic scenarios, addressing key issues such as dynamic spatial sound field modeling and simulation, multi-source acoustic perception, spatial audio generation, and robust speech processing. It emphasizes the integration of physical models and data-driven methods, multimodal information collaborative modeling, and the development of spatial audio foundation models, promoting deep cross-disciplinary integration among array signal processing, speech processing, artificial intelligence, and related application fields.

This session aims to build a platform for exchange and cooperation among experts and scholars in relevant domestic fields, promote the development of key technologies such as dynamic sound field modeling, spatial audio perception, and robust speech processing, and provide theoretical foundations and key technical support for a new generation of intelligent interaction applications including robots, intelligent terminals, intelligent meetings, digital humans, and embodied intelligence. The session holds important scientific significance and application value: at the scientific level, the "physical mechanism + data-driven" fusion paradigm explored in this session can effectively compensate for the shortcomings of deep learning methods in interpretability and physical consistency, opening new paths for the cross-disciplinary integration of computational acoustics and artificial intelligence; at the application level, these core technologies are of great significance for meeting urgent needs in key fields such as defense communications, smart homes, healthcare, and embodied intelligence, and can also provide valuable references for the development and continuous progress of related technologies in other fields.

Yanmin Qian
Photo
Yanmin Qian
Shanghai Jiao Tong University
Gongping Huang
Photo
Gongping Huang
Wuhan University
Wangyou Zhang
Photo
Wangyou Zhang
Shanghai Jiao Tong University
Zhongxin Bai
Photo
Zhongxin Bai
Wuhan University
▌ Challenges in Multichannel Acoustic Signal Perception and Processing in Complex Environments
Jingdong Chen · Wuhan University · Professor · jingdongchen@ieee.org

Multichannel acoustic signal processing systems, including two-channel systems, microphone arrays, loudspeaker arrays, MIMO systems, and distributed acoustic networks, are widely used in acoustic signal sensing and acquisition, separation and enhancement, event detection and recognition, semantic understanding, and sound field reconstruction. With the development of sensors, signal processing, artificial intelligence, and communication technologies, related methods have been continuously innovated, driving improvements in the performance and application capabilities of multichannel acoustic signal processing. However, in complex acoustic environments, factors such as noise, reverberation, coexistence of multiple sound sources, channel discrepancies, and resource constraints still pose significant challenges to high-fidelity acquisition, efficient transmission, robust processing, and accurate reconstruction of acoustic signals. How to fully exploit the spatial-temporal characteristics of signals to achieve efficient sensing, collaborative processing, and high-fidelity sound field perception and reconstruction remains a critical open problem. This talk will review research progress in multichannel acoustic signal perception and processing, analyze key technical issues and major challenges in complex environments, and discuss future development directions.

▌ Audio-Visual Fusion for Spatial Collaborative Perception: Techniques and Applications
Junfeng Li · Institute of Acoustics, Chinese Academy of Sciences · Professor · lijunfeng@hccl.ioa.ac.cn

When perceiving the complex real world, human vision and hearing are highly collaborative. With the rapid development of multimedia technology and embodied intelligence, cross-modal audio intelligence (particularly audio-visual fusion technology) has become a frontier hotspot in the field of artificial intelligence. Audio-visual fusion perception aims to break the incompleteness of single modalities and achieve precise localization of sounding targets and environmental understanding in complex scenes by exploiting the strong correlation and complementary information between the two modalities. This is not only a key step for machines to build human-like perceptual capabilities, but also shows broad application prospects in intelligent video surveillance, natural human-computer interaction, and autonomous robot navigation. However, in real open complex environments, how to effectively utilize vision and hearing to achieve deep collaboration and overcome challenges such as reverberation, multiple sound sources, and inter-modal information asymmetry remains a difficult research problem. This talk will focus on the core task of audio-visual fusion-based spatial collaborative perception, systematically review the latest evolution of this field, discuss the underlying logic of audio-visual joint perception, the technical paradigms of mainstream frameworks in multimodal feature representation and spatial-temporal information fusion, and look ahead to the new opportunities, potential challenges, and future directions of cross-modal perception.

▌ From Acoustic Perception to Sound Field Control: Progress and Prospects of Acoustic Array Technology
Jing Lu · Nanjing University · Professor · lujing@nju.edu.cn

Acoustic information acquisition and processing based on microphone arrays is not only an important front-end technology for speech communication and human-computer interaction systems, but also provides key acoustic information and feedback for sound field control. The development of acoustic array technology, on the one hand, continuously improves the ability of acoustic information acquisition and understanding in complex environments; on the other hand, it continuously expands the capability of active sound field control, providing important support for high-quality speech interaction, low-noise environment construction, personalized listening, and immersive sound field experiences. This talk will introduce the recent research progress and industrial collaboration achievements of the Audio and Acoustics Laboratory at Nanjing University in the fields of acoustic perception and sound field control, and discuss the new opportunities and future directions of acoustic array technology in the context of the development of artificial intelligence.

▌ Extraction Methods for Dynamic Sound Sources in Interfering Environments
Chao Pan · Northwestern Polytechnical University · Associate Professor · cpan@nwpu.edu.cn

In complex acoustic scenes, prior information is difficult to obtain for moving or sudden dynamic sound sources, and traditional microphone array beamforming and filtering methods are prone to performance degradation or even failure. This paper proposes a signal extraction framework with array self-sensing capability based on the covariance matrix residual model. Given the coherence matrices of interference and noise, a two-stage matrix decomposition is performed to separate the residual component from the observed array covariance matrix, estimating the coherence matrix and power of unknown dynamic sound sources, thereby enabling real-time detection and signal extraction of newly appearing unknown sources. The method requires no priors such as the direction or characteristics of the target source, can handle transient and time-varying sources, and exhibits good robustness even when the number of interferences is misestimated. Compared with classic blind source separation algorithms, this scheme achieves significant SNR improvement, offering a new approach for multichannel signal processing in complex sound fields.

▌ Mixture of Experts for Universal Sound Separation: Acoustic and Semantic Guided Sparse Routing
Chenda Li · Shanghai Jiao Tong University · Assistant Researcher · lichenda1996@sjtu.edu.cn

Universal sound separation aims to recover individual sources from mixtures containing speech, singing, music, and environmental sounds. The significant differences among sounds in spectral structure and temporal variation impose high demands on model expressiveness and create a trade-off between model capacity and computational cost. This talk introduces a sparse mixture-of-experts method for universal sound separation, which expands model capacity while controlling computational cost through two complementary routing mechanisms. The time-frequency-axis routing extends expert routing strategies from speech separation to universal sound separation, selecting experts based on acoustic features of different time frames and frequency bands; the event-supervised mixture-level routing uses source event labels during training to guide expert selection, so that the event information of the mixture and local acoustic features complement each other. At inference time, the model relies solely on the mixture audio, completing multi-source separation in a single forward pass without event labels or textual prompts. The method achieves an 11.9 dB signal-to-distortion ratio improvement (SDRi) on the Hive three-source separation test set, outperforming baselines while reducing multiply-accumulate operations by 84.8%. The talk will present the design of the two routing mechanisms with separation examples, experimental results, and routing visualizations, and discuss the impact of training data scale and model capacity on separation performance.

Yanmin Qian (Shanghai Jiao Tong University): yanminqian@sjtu.edu.cn

Gongping Huang (Wuhan University): gongpinghuang@whu.edu.cn

Wangyou Zhang (Shanghai Jiao Tong University): wyz-97@sjtu.edu.cn

Zhongxin Bai (Wuhan University): 18392387962@163.com