Speaker tracking is the task of separating and following multiple speakers over time. It must address overlapped speech, speech onset and offset, talker identity, and time varying speaker count. We propose a segment-online, modular framework for single- and multi-channel speaker tracking. The proposed system first performs speaker separation in each segment and computes speech activity via voice activity detection. To generate speaker tracks over time, we introduce a two-stage sequential organization strategy: Overlap-based stitching for continuous grouping and memory-based speaker verification for discontinuous grouping. For speaker separation, we employ complex spectral mapping to estimate the real and imaginary spectrograms of underlying speakers. The proposed system achieves state-of-the-art segment-online tracking performance on the LibriCSS and AMI datasets. Our framework significantly reduces diarization error rate (DER) and concatenated minimum-permutation word error rate (cpWER) compared to other methods.

DeLiang Wang received the B.S. degree in 1983 and the M.S. degree in 1986 from Peking (Beijing) University, and the Ph.D. degree in 1991 from the University of Southern California, all in computer science. From 1991 to 2025, he was with the Department of Computer Science and Engineering and the Center for Cognitive and Brain Sciences, Ohio State University, where he was a Professor and University Distinguished Scholar. Since July 2025, he has been with the School of Data Science, the Chinese University of Hong Kong, Shenzhen, where he is a Presidential Chair Professor. His research interests include machine perception and deep learning. He was the recipient of the 1996 Office of Naval Research Young Investigator Award, and the 2008 Helmholtz Award from the International Neural Network Society, and the 2025 Neural Networks Pioneer Award from the IEEE Computational Intelligence Society. He also received the 2007 Outstanding Paper Award from the IEEE Computational Intelligence Society, the 2019 Best Paper Award and 2025 Donald G. Fink Overview Paper Award from the IEEE Signal Processing Society. He is a Fellow of IEEE, ISCA and AAIA, and served as the Editor-in-Chief of Neural Networks from 2011 to 2025.