每日 arXiv

音频 · 语音 · 音乐 · 声学

每日新爬取的 arXiv 论文，按评分排序。中文摘要帮你快速判断是否值得深读。

2026-08-12
日期2026-08-12
已评分—
均分—
最高—

Daily Papers — 2026-08-12

17 papers on audio, speech, music, and acoustics.

1. Do Text-to-Music Models Really Follow Instructions? A Counterfactual Evaluation of Key and Beat Grouping

Authors: Yining Wang

Categories: cs.SD | 18 pages, 3 figures Score: 8.40/10 (Obj:9 Id:9 Ind:9 Comp:9 Eff:9 Nov:9 Rep:7)

  • Strength: Matched counterfactual 框架用 neutral baseline + target-swap 将”目标出现”与”指令可归因控制”分离，揭示 Stable Audio 3 四拍 agreement 从 neutral 0.969 降至 treatment 0.563 的反直觉发现，且经 placebo（+0.003~+0.008）、5-rater 盲评（family-level 相关 0.80–0.84）、3-see。
  • Weakness: 全矩阵仅单 deterministic seed，ACE-Step 缺 contemporaneous checkpoint pin，评测仅覆盖 global key 和 3/4-beat grouping 两个属性、三个系统，且 human audit 仅含单 bridge rater 的 28-family 子集，无法支持 per-model 人类排名。
  • Full review: Claude Code 全文七公理审稿

Prompted attribute agreement is widely used as evidence of text-to-music controllability, yet a requested attribute may occur simply because it is already common in the model’s output distribution. We introduce a matched counterfactual evaluation that separates target occurrence from instruction-attributable control. Each family contains a neutral input that omits the scored attribute and two otherwise matched inputs that swap the requested target. All three are rendered through frozen native-interface adapters with a shared seed. Applied to global key and beat grouping in three open systems, this design changes the empirical conclusion. ACE-Step 1.5 and Stable Audio 3 Medium exhibit substantial key control, whereas LeVo2 does not. For beat grouping, the same models redirect toward the rare three-beat target, but high four-beat agreement is largely inherited from neutral outputs: Stable Audio 3 produces four-beat grouping in 0.97 of neutral cases but only 0.56 under its explicit four-beat treatment. Off-attribute placebos, external recognizer validation, blind expert annotation, and multi-seed sentinels support the attribution. When targets have unequal output priors, agreement describes what a model produced, while matched neutral and target-swap contrasts test whether the instruction changed it.

2. On-Policy Self-Distillation for Multi-Dialect ASR: Mastering Dialects, Retaining Mandarin

Authors: Shuiyuan Wang, Bingshen Mu, Pengshen Zhang, Chengyou Wang, Yujie Liao et al.

Categories: eess.AS Score: 6.95/10 (Obj:8 Id:8 Ind:6 Comp:5 Eff:8 Nov:5 Rep:8)

  • Strength: 在匹配数据与训练计划的控制实验中，OPSD 将普通话平均 CER 从 3.40% 降至 3.27%、方言平均 CER 从 13.16% 降至 12.79%，而 Continued SFT 反使普通话 CER 升至 4.43%，干净地验证了 distillation 在保护已有能力方面的优势。
  • Weakness: OPSD 本质是 LLM 领域 on-policy distillation 向 ASR 的跨领域迁移，但论文缺少 privileged context 消融、scheduled sampling 对比及 teacher guidance 方式的 ablation，组件必要性与方法新颖性论证不足。
  • Full review: Claude Code 全文七公理审稿

Recent large-scale ASR models already achieve strong Mandarin recognition accuracy and have some ability to recognize Chinese dialects. However, their dialect recognition accuracy is still limited in real-world speech. Direct dialect adaptation can lower dialect CER, but it may also raise Mandarin CER. We therefore study how to adapt a capable ASR model to improve multi-dialect recognition without degrading Mandarin recognition. We adopt an adaptation pipeline where continual pre-training (CPT) and dialect supervised fine-tuning (SFT) provide a strong foundation, and On-Policy Self-Distillation (OPSD) serves as the final refinement. OPSD addresses the train–test mismatch in autoregressive ASR by training the student model on its own decoded prefixes while a frozen teacher, conditioned on the reference transcript as privileged context, provides soft token-level targets. This replaces hard cross-entropy updates on dialect data with distillation, preserving Mandarin ability while refining dialect recognition. We instantiate the framework with Qwen3-ASR-1.7B and evaluate it on public and internal Mandarin and dialect test sets. Under matched refinement data and schedule, OPSD improves dialect recognition without raising Mandarin CER, whereas continued teacher-forced fine-tuning increases Mandarin CER. We will release the model weights and evaluation scripts.

3. Luna-TTS Family Technical Report

Authors: Feng Yin, Shuai Shi, Junjie Zheng, Kechenying Zhou, Yiqiu Wang et al.

Categories: cs.SD, eess.AS Score: 6.60/10 (Obj:9 Id:5 Ind:5 Comp:8 Eff:8 Nov:6 Rep:5)

  • Strength: 在 1M 小时四语言数据上构建的 diffusion-LLM TTS 系统，Seed-TTS-Eval 四项指标全面领先 AR SOTA (0.73 CER / 79.7 SIM / 1.49 WER / 76.8 SIM)，RTF 0.0211 和 41.6ms 首块延迟均为对比中最优。
  • Weakness: 缺少同 backbone/同数据的纯 AR 对照实验和 RL 前后消融，无法将 SOTA 表现归因于 diffusion 范式本身而非数据规模或训练工程；Gemini 3.1 Pro 同时用于训练标注和评测 judge 构成循环论证风险。
  • Full review: Claude Code 全文七公理审稿

Modern text-to-speech (TTS) is dominated by autoregressive (AR) codec language models, whose left-to-right decoding brings latency that grows with utterance length, error accumulation along the committed prefix, and an artificial generation order imposed on the Residual Vector Quantization (RVQ) token grid. We propose Luna-TTS Family, diffusion-language-model-based TTS systems pretrained on 1 million hours of speech across Chinese, English, Japanese, and Korean. The family is built by progressive adaptation of a pretrained AR text LLM, from causal to bidirectional and finally to block-causal attention, and comprises two variants sharing a single tokenizer, data pipeline, and 0.6B backbone lineage. Luna-TTS is fully non-autoregressive: it generates the entire RVQ token grid in a fixed number of parallel refinement steps, with zero-shot voice cloning and speech editing arising natively as infilling. Luna-TTS Realtime, derived by continual training, is autoregressive over blocks of 32 codec frames (1.28s) while denoising each block in parallel; it supports KV-cached blockwise generation and incremental audio delivery, achieving an end-to-end RTF of 0.0240 and 41.6 ms local first-block latency under the warmed serving protocol. An annealed fine-tuning stage adds explicit control over emotion and non-verbal vocalizations (NVVs), and a reinforcement-learning stage applies GRPO with policy ratios computed over the realized denoising trajectory. On Seed-TTS-Eval, Luna-TTS achieves the best results on all four metrics among compared open-source and commercial systems (0.73 CER / 79.7 SIM on test-zh, 1.49 WER / 76.8 SIM on test-en); on the harder in-the-wild CV3-Eval, it posts the lowest Mandarin and English error rates in our comparison. Against leading commercial systems, it achieves the best results on most objective, model-based, and human-rated metrics for NVV and emotion control.

4. The SLT 2026 SmartGlasses Challenge: Benchmarking Egocentric Multi-Talker Speech Recognition and Understanding with Audio-Language Models

Authors: Dehui Gao, Zhixian Zhao, Zhennan Lin, Yujie Liao, Yuhang Dai et al.

Categories: eess.AS, cs.SD | 7 pages, 7 figures Score: 6.58/10 (Obj:7 Id:5 Ind:6 Comp:6 Eff:6 Nov:5 Rep:8)

  • Strength: 引入 106 小时四通道自我中心普通话语音数据集与联合 TSA-ASR+SLU 基准，15 队提交结果表明 Track 1 最优 tcpCER 5.23%（可用级），而重叠率分层分析提供了可迁移的经验证据。
  • Weakness: 核心发现（重叠损害 ASR、声学题更难）属预期内确认而非新洞察；基线覆盖仅 2 个且缺少最简 baseline，系统设计选择缺乏受控消融，SLU 的 MCQ 格式与 LLM 辅助出题流程存在独立性隐患。
  • Full review: Claude Code 全文七公理审稿

Recent advances in large language models (LLMs) and multimodal LLMs (MLLMs) have created new opportunities for wearable speech interfaces, with smart glasses providing an egocentric platform for continuous audio sensing and assistance. However, speech recognition and understanding in this setting remain challenging because of dynamic acoustic conditions, speaker overlap, and the spatial ambiguity introduced by wearer-centered recording geometry. To support systematic evaluation in this setting, we introduce the IEEE SLT 2026 SmartGlasses Challenge for egocentric multi-speaker speech processing. The challenge consists of two tracks, Dyadic Dialogue Understanding and Multi-party Meeting Understanding, and jointly evaluates Time-Stamped Speaker-Attributed Automatic Speech Recognition (TSA-ASR) and Spoken Language Understanding (SLU). It is built on a 106-hour four-channel egocentric speech dataset containing 714 sessions collected in real-world scenarios. This paper describes challenge tasks, dataset construction, submissions, and summarizes the main findings from the shared evaluation. The results show that heavy speaker overlap remains a major factor affecting TSA-ASR performance, while paralinguistic acoustic understanding continues to be difficult for current audio-language models in complex SLU settings. Further details can be found on the official challenge website.

5. ResPCC: A Loss-Resilient Neural Point Cloud Codec over Lossy Networks

Authors: Xueqin Niu, Mufan Liu, Yifan Wang, Le Yang, Jun Sun et al.

Categories: eess.IV, cs.MM Score: 6.20/10 (Obj:8 Id:6 Ind:8 Comp:5 Eff:7 Nov:5 Rep:8)

  • Strength: 首个端到端丢包鲁棒神经点云编解码器，在 ShapeNet/SemanticKITTI 上 5%-30% 丢包率下相比 DPCC 提升 14-23% D1/D2-PSNR，四模块协同设计有效且代码开源。
  • Weakness: 四个核心模块（CALM/SCI/MGLR/DBR）均为成熟技术迁移组合（FiLM/interleaver/GNN-imputation/VQ-refinement），缺乏理论 justification 和最简 baseline 对比，backbone 同时改进导致变量隔离不彻底。
  • Full review: Claude Code 全文七公理审稿

Point cloud compression (PCC) is critical for efficient storage and transmission of 3D data. While recent learning-based PCC methods achieve good rate-distortion (R-D) performance, they generally rely on ideal transmission conditions. In practice, packet loss is a common issue and can severely distort latent features, causing coordinate drift and geometric degradation. To address this challenge, we present ResPCC, the first end-to-end neural point cloud codec designed to offer intrinsic resilience against data loss. Our framework is loss-rate-aware and adapts to diverse packet loss conditions. At the encoder, we introduce a Condition-Adaptive Latent Modulation (CALM) module to adjust latent feature distributions according to the perceived loss rate, as well as a Spatial-Channel Interleaving (SCI) mechanism that transforms channel-wise data extinction into spatially scattered element-wise missing patterns. At the decoder, we develop a Mask-Aware Graph-based Latent Restoration (MGLR) module, followed by a Dictionary-based Refinement (DBR) stage to recover corrupted features and align them with canonical priors. Evaluations on ShapeNet and SemanticKITTI under 5\% to 30\% packet loss rates show that ResPCC consistently delivers superior stability and R-D performance over baselines. Our framework maintains high reconstruction fidelity under lossy conditions, providing a reliable solution for 3D data transmission over practical networks. Code is available at https://github.com/starrynight314/ResPCC.

6. Dialogue-Aware Video-to-Music Generation Using Public Domain Film Collections

Authors: Haven Kim, Zachary Novack, Julian McAuley, Hao-Wen Dong

Categories: cs.SD, cs.CV, cs.MM Score: 5.95/10 (Obj:5 Id:5 Ind:8 Comp:8 Eff:4 Nov:5 Rep:8)

  • Strength: OSSL-v2 提供了 246.4h/34,343 clips 的自托管公共领域电影配乐数据集，解决了 V2M 领域长期的数据复现问题；在统一数据上对三个 SOTA 模型进行了首次公平 benchmark。
  • Weakness: Dialogue adapter 对最强模型 GVMGen 在 in-distribution 上导致净负面效果（CLAP 0.43→0.39, Precision 0.56→0.42），且所有模型的绝对指标偏低（CLAP < 0.45, Precision 多处为 0.00），缺乏人类主观评测。
  • Full review: Claude Code 全文七公理审稿

Video-to-music generation has drawn growing interest for its role in conveying the emotion of visual media, including film. Progress in the field, however, is hampered by a reproducibility gap: models are often trained on crawled corpora referenced through YouTube URLs that may be deleted, with the underlying data often difficult and time-consuming to retrieve. To address this, we introduce the Open Screen Soundtrack Library version 2 (OSSL-v2), a self-hosted corpus of 34,343 video clips totaling 246.4 hours, sourced from public-domain films. Unlike crawled corpora, OSSL-v2 is reproducible (i.e., not subject to link rot) and copyright-conscious, yet still large enough to train functional video-to-music models. We then use this film-domain corpus to study dialogue as a conditioning signal for video-to-music generation, motivated by the close temporal coupling between film music and on-screen speech. Specifically, we augment existing models’ video cross-attention with a time axis and modulate it frame-by-frame with the dialogue track. Evaluated on both public-domain and commercial films, our approach shows improvement over the state-of-the-art baselines. The dataset is available at https://huggingface.co/datasets/McAuley-Lab/OSSL-v2.

7. Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encoder

Authors: Huaxuan Wang, Huimin Wang, Ruiyu Zhang, Yingjie Li, Yitao Duan

Categories: cs.SD, cs.CL | 12 pages, 1 figure, 6 tables Score: 5.95/10 (Obj:8 Id:5 Ind:6 Comp:5 Eff:8 Nov:4 Rep:9)

  • Strength: 在 CV3-Eval 跨语言 benchmark 上取得 3.73% 平均 WER，6 方向中 4 方向最优，人工评测 4 方向中 3 方向 overall rank 第一，且开源代码与 checkpoint。
  • Weakness: 缺乏任何 ablation 实验，核心技术（learnable speaker encoder on SSL features）与 MiniMax-Speech 和 Qwen3-TTS 高度重叠，新颖性增量有限。
  • Full review: Claude Code 全文七公理审稿

Recent advances in zero-shot text-to-speech (TTS) have substantially improved speech quality and voice cloning fidelity. However, many zero-shot TTS systems still depend on audio prompt transcripts at inference time. This dependency limits cross-lingual voice cloning, since in-the-wild reference audio is often untranscribed. In this technical report, we present Confucius4-TTS, a multilingual zero-shot TTS system that supports 14 languages and performs both intra-lingual and cross-lingual reference cloning without requiring transcripts of audio prompts. Confucius4-TTS follows a two-stage architecture, consisting of text-to-semantic (T2S) and semantic-to-acoustic (S2A) modules. The LLM-based T2S module uses a learnable speaker encoder to extract timbre features from self-supervised speech representations, and the conditional flow-matching S2A module converts the predicted semantic tokens into mel-spectrograms. The same model also supports continuation cloning when a reference transcript is available. Confucius4-TTS is trained on large-scale multilingual speech data. It achieves high intelligibility and speaker similarity on public benchmarks. On the CV3-Eval cross-lingual benchmark, Confucius4-TTS obtains an average WER of 3.73% across six directions. On our internal cross-lingual set, it achieves the best average overall rank in human evaluation among recent open-source and commercial systems. We release code, model checkpoints, and demos at https://github.com/netease-youdao/Confucius4-TTS.

8. MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching

Authors: Xingwei Sun, Heinrich Dinkel, Gang Li, Jiahao Mei, Yadong Niu et al.

Categories: eess.AS, cs.SD Score: 5.90/10 (Obj:7 Id:5 Ind:6 Comp:5 Eff:6 Nov:4 Rep:8)

  • Strength: 统一音频场景生成中语音可懂度大幅提升（Seed-TTS EN WER 从 12.15% 降至 2.79%），接近专用 TTS 系统水平，且开源代码和模型 checkpoint。
  • Weakness: 核心方法（AR per-token flow matching）与 FELLE (2025-02) 和 AudioCALM (2026-06) 高度重叠，缺少关键 baseline 比较（UniSonate WER 1.47% 更优），且未隔离 LLM backbone、额外 TTS 训练数据和 AR flow matching 各自的贡献。
  • Full review: Claude Code 全文七公理审稿

Generating coherent audio scenes that simultaneously blend speech, music, and sound effects remains a significant challenge. Current approaches typically rely on a disjointed pipeline where a frozen, decoupled text encoder feeds a separate audio decoder, limiting cross-modal optimization and leading to poor speech intelligibility. To overcome these limitations, we introduce MiDashengLM-Gen, an end-to-end framework that couples a pre-trained Large Language Model (LLM) with per-token conditional flow matching for autoregressive, variable-length mixed-audio scene generation. MiDashengLM-Gen represents a first approach for general text-to-audio generation with one end-to-end trained model. Empirical evaluations demonstrate that MiDashengLM-Gen drastically improves speech intelligibility over existing unified models. On the Seed-TTS benchmark, English Word Error Rate (WER) drops from 12.15% to 2.79%, approaching the performance of dedicated Text-to-Speech (TTS) systems (1.24%). Furthermore, the framework extends effectively to multilingual settings, yielding highly competitive multilingual WERs compared to existing baselines. Lastly, the model maintains competitive mixed-audio generation quality on the MECAT benchmark. Code and checkpoints are available at https://github.com/xiaomi-research/midashenglm-gen and https://huggingface.co/mispeech/midashenglm-gen, and the demo page is available at https://xingws.github.io/midashenglm-gen-demo/.

9. UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos

Authors: Yuxuan Zhang, Haozhong Xiong, Jiayi Song, Jinpeng Yu, Yang Shi et al.

Categories: cs.CV, cs.SD Score: 5.84/10 (Obj:8 Id:5 Ind:6 Comp:5 Eff:6 Nov:6 Rep:5)

  • Strength: 首次将 talking video 的外观+声音身份替换统一到单个 AV diffusion transformer 中流式生成，Sync-C 3.633 超过级联最强 baseline 3.289，长视频 DINO-S 在 60s 内保持 0.590-0.596 无退化，Feature-RoPE Decomposition 消融验证了三组件必要性。
  • Weakness: 视频质量 (ASE 2.097, IQA 3.758) 和声音质量 (BAK 3.563, SECS 0.730) 多项指标落后于分别优化的 baseline，Efficient Multi-LoRA Switching 和 swap-and-reconstruct pipeline 缺少消融，最直接相关工作 OmniForcing (ECCV 2026 Oral) 未作为实验 baseline。
  • Full review: Claude Code 全文七公理审稿

Talking-video character replacement requires coordinated transfer of appearance and voice while preserving the source motion, scene, linguistic content, and audio-video timing. Existing methods use separately optimized models for the two modalities, making audio-visual consistency difficult to enforce. We present UniSwap, the first framework for streaming joint audio-visual identity replacement in talking videos. Given a source video, a reference image, and a reference voice clip, UniSwap transfers the reference appearance and vocal timbre within a single audio-visual diffusion transformer while preserving the source content and dynamics. To address the scarcity of aligned cross-identity training pairs, we introduce a swap-and-reconstruct pipeline that removes visual and vocal identity from real clips and uses the original clips as reconstruction targets. Starting from a bidirectional backbone, we progressively adapt the model through In-context Pretraining for joint replacement, Conditional Streaming Adaptation for block-causal KV-cached generation, and Efficient Self-forcing DMD for mitigating exposure bias and reducing sampling from 30 to 3 denoising steps per block. Efficient Multi-LoRA Switching enables the three DMD roles to share a single frozen backbone. Feature-RoPE Decomposition keeps cached positions within the training range, supporting stable long-form inference. Experiments demonstrate strong audio-visual synchronization, competitive identity preservation, efficient streaming, and stable long-form generation.

10. RT-SEMamba: Real-Time Speech Enhancement Mamba via Progressive Knowledge Distillation

Authors: Rong Chao, Sung-Feng Huang, Moreno La Quatra, Sabato Marco Siniscalchi, Wen-Huang Cheng et al.

Categories: cs.SD, cs.CL | Accepted to INTERSPEECH 2026 Score: 5.47/10 (Obj:8 Id:5 Ind:8 Comp:5 Eff:5 Nov:4 Rep:8)

  • Strength: 在严格 25 ms algorithmic latency 约束下，8-layer causal Mamba 达到 3.32 PESQ，并通过 KD 将 1-layer student 从 3.06 提升至 3.18 PESQ 且不增加推理开销。
  • Weakness: 核心方法是 causal SEMamba + 标准 KD 的 A+B 组合，缺少与同期流式 SE 工作 (FastEnhancer, LaCo-SENet) 的比较，KD 各组件 ablation 不充分，且 Table 3 的跨模型比较因训练语料和延迟差异不公平。
  • Full review: Claude Code 全文七公理审稿

We present RT-SEMamba, a fully causal speech enhancement (SE) model built upon causal time-frequency Mamba blocks. Unlike Transformer-based architectures that rely on a growing key-value cache, Mamba propagates a fixed-size recurrent state per layer, enabling memory- and bandwidth-efficient long-form inference. We further introduce a progressive knowledge distillation (KD) strategy that compresses an 8-layer teacher into a shallow 1-layer student by jointly distilling complex spectral outputs and intermediate representations. On Voicebank-DEMAND, the 8-layer RT-SEMamba achieves 3.32 PESQ with a 25 ms algorithmic latency constraint, and the distilled 1-layer student improves over a naive 1-layer baseline from 3.06 to 3.18 PESQ while preserving the same steady-state RTF, delivering a 2.75x speedup over the teacher. These results demonstrate that state-space models with progressive KD provide a competitive quality-latency trade-off for real-time SE.

11. MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques

Authors: Jiabao Zhuang, Changhao Jiang, Hanchen Wang, Jiahao Chen, Zhixiong Yang et al.

Categories: cs.SD, cs.CL Score: 5.30/10 (Obj:7 Id:5 Ind:5 Comp:5 Eff:5 Nov:4 Rep:8)

  • Strength: 两阶段评论条件化奖励模型在 SongEval 上将 MSE 从 0.2875 降至 0.2316，五维 LCC 达 0.9068，且消融实验清晰隔离了评论、自生成评论和 SFT 初始化各自的贡献。
  • Weakness: 核心方法直接迁移自 Critic-RM（NAACL 2025）和 CLoud（2024），OOD 提升（71.35% vs 70.80%）在统计噪声内，GRPO 评估使用训练数据源 SongEval 作为评估器存在循环性，且缺少 HEAR 等近期 baseline 比较。
  • Full review: Claude Code 全文七公理审稿

Long-form song generation models continue to improve in duration, structural integrity, and acoustic complexity, making reliable aesthetic rewards increasingly important for aligning these models with human preferences. However, reward models for complete songs remain limited, and existing evaluators typically predict scores in a single forward pass without providing readable explanations. We introduce MUSECRITIC, a semi-scalar reward model that generates a natural-language critique covering five aesthetic dimensions and uses it as an intermediate representation to predict continuous reward scores. MUSECRITIC follows a two-stage training pipeline: a teacher model first provides high-quality critiques for supervised fine-tuning, after which the fine-tuned model generates its own critiques for reward learning, mitigating distribution shift between training and inference. On an in-domain test set of 200 SongEval songs, MUSECRITIC reduces macro-averaged mean squared error from 0.2875 to 0.2316 and improves macro-averaged LCC, SRCC, and Kendall’s tau to 0.9068, 0.8838, and 0.7178, respectively. On the out-of-domain Music Arena benchmark with 733 preference pairs, it achieves the highest accuracy of 71.35%. Moreover, using MUSECRITIC with GRPO improves Muse-0.6B on all nine aesthetic metrics from SongEval and Audiobox Aesthetics. These results demonstrate that critique-conditioned reward modeling reduces scoring error and provides an effective optimization signal for song generation. The project repository is available at https://github.com/WuqnEl/MuseCritic.

12. Deep Learning Based Relative Transfer Matrix Estimation for Multiple Sources and Multiple Microphones

Authors: Oshan A. B. Yalegama, Wageesha N. Manamperi

Categories: eess.AS, cs.AI, eess.SP | Accepted to Interspeech 2026 Score: 5.26/10 (Obj:8 Id:4 Ind:5 Comp:6 Eff:5 Nov:5 Rep:5)

  • Strength: 首次将深度学习用于 ReTM 估计；FuSNet 在主要场景中将 SDR 比基于协方差的基线提高了高达 17 dB（场景 C：40.33 vs 23.05 dB）。
  • Weakness: 无消融实验，仅一个基线，单一模拟房间，且 FuSNet 的语音增强在场景 C 中导致 SDR 为负（-1.19 dB），性能比噪声输入还差。
  • Full review: Claude Code 全文七公理审稿

The Relative Transfer Matrix (ReTM), recently introduced as a generalization of the relative transfer function for multiple receivers and sources, shows promising performance when applied to speech enhancement in noisy environments. Estimating the ReTM of sound sources by exploiting the covariance matrices of multichannel recordings is highly beneficial for practical applications and, to date, remains the only proposed approach. This paper investigates deep learning-based ReTM estimation. We propose three novel supervised learning frameworks using time and short-time frequency transform domain convolutional networks, and a Long Short-Term Memory-based recurrent neural network. Experimental results demonstrate that the proposed models achieve more accurate estimation of the ReTM using five objective metrics compared to the covariance-based method. We also show the effectiveness of the proposed frameworks for speech enhancement, achieving performance on par with the baseline method.

13. Easper: An Accessible ASR Pipeline for Language Documentation

Authors: Aso Mahmudi, Ting Dang, Ekaterina Vylomova, Nick Thieberger

Categories: cs.CL | Accepted in Interspeech 2026 Score: 5.26/10 (Obj:8 Id:5 Ind:6 Comp:5 Eff:5 Nov:4 Rep:8)

  • Strength: 在三种 Vanuatu 语言上首次实证证明，优先转录词汇重复丰富的会话（ToTy 策略）比优先选择声学干净录音更能加速 Whisper 微调的 CER 下降，为语言文档记录提供了可操作的数据收集指导。
  • Weakness: 未报告任何具体 CER 数值或统计显著性检验，未与 ASR 主动学习领域的标准基线比较，系统贡献（Easper）与已有工具 Elpis 的增量有限且未在同一实验中对比。
  • Full review: Claude Code 全文七公理审稿

Audio transcription is a critical bottleneck in language documentation. While multilingual Automatic Speech Recognition (ASR) models like Whisper offer solutions, field linguists often lack the expertise to utilise them. We present Easper, an open-source, no-code workflow enabling linguists to iteratively fine-tune ASR models via cloud resources directly from ELAN annotations. Deploying ASR also raises a cold start problem: deciding which recordings to transcribe first to bootstrap an accurate model. Using Easper, we evaluate transcription prioritisation strategies on three Vanuatu languages (Bislama, Nafsan, Nguna). We fine-tune models by recording session, comparing Character Error Rate trajectories when prioritising acoustic cleanliness versus linguistic richness. We demonstrate that prioritising lexically rich narratives and increasing acoustic-phonetic repetition, even in noisy environments, leads to faster improvements in transcription quality.

14. Phoenix TTS: High-Fidelity Synthesis and Voice Conversion via Flow-Matching-Driven Speech Tokenization

Authors: Peijie Chen, Zhuanling Zha, Zhipeng Nie, Weijie Wu, Yiming Liu et al.

Categories: cs.SD Score: 5.05/10 (Obj:8 Id:6 Ind:6 Comp:5 Eff:8 Nov:3 Rep:5)

  • Strength: 在仅 110K 小时训练数据上实现 WER 低于 GT、SIM 达到或超过 GT 的零样本 TTS 性能，9 个 baseline 对比中 4/6 SIM 指标和 3/3 WER 指标最佳或接近最佳，主观评测 SMOS/CMOS 均为最优。
  • Weakness: 核心 novelty（tokenizer 与生成模型联合优化）与 SiTok (2602.06602, 2026年2月) 高度重叠且未引用，其余组件均为已有方法拼装，代码未开源，缺少与最相似方法 SiTok 的直接比较。
  • Full review: Claude Code 全文七公理审稿

In current zero-shot text-to-speech systems, conventional semantic tokenizers are typically optimized using supervised automatic speech recognition or self-supervised learning objectives. However, due to the inherent nature of speech, semantic and acoustic information cannot be completely decoupled, and ASR-based tokenizers discard acoustic details to focus on linguistic content; models relying on them usually struggle to achieve optimal speaker similarity. Furthermore, these tokenizers are optimized independently and lack direct supervision from downstream acoustic generation tasks. This isolated training creates a feature gap between the extracted discrete tokens and the continuous space required by acoustic models, fundamentally bottlenecking the upper bound of synthesis quality. To bridge this gap, we propose Phoenix TTS, a unified framework that tightly couples representation learning with generative acoustic modeling. Specifically, our speech tokenizer is optimized to reconstruct self-supervised features to maintain semantic richness, while simultaneously receiving direct supervision from a Flow Matching training loss. Through this joint training paradigm, the extracted discrete tokens successfully preserve essential semantic information and natively align with the feature space of the downstream Flow Matching model. Comprehensive evaluations highlight the efficiency and effectiveness of Phoenix TTS. Trained on 110K hours of data, the system achieves excellent speech intelligibility, yielding WER that consistently falls below that of ground-truth recordings. Simultaneously, it maintains robust zero-shot speaker similarity, rivaling or outperforming several prominent large-scale baselines. Furthermore, as an advantageous byproduct of this unified training, the learned tokenizer can be seamlessly adapted to zero-shot voice conversion tasks without requiring task-specific fine-tuning.

15. HarmoniDPO: Video-guided Audio Generation via Preference-Optimized Diffusion

Authors: Wenshuo Peng, Kaipeng Zhang

Categories: cs.CV | 31 pages Score: 4.95/10 (Obj:8 Id:5 Ind:4 Comp:5 Eff:5 Nov:4 Rep:5)

  • Strength: 在 VGGSound 和 AVSync15 上全面超越 FoleyCrafter/Frieren 等 baseline（MKL 1.82 vs. 2.56, CLIP 13.65 vs. 10.70, FID 6.42 vs. 12.48），dual video representation + online-DPO + DDS 三阶段递进提升清晰。
  • Weakness: 核心 novelty claim（”首次将 DPO 应用于 V2A”）被 V2A-DPO (2026年3月) 抢先5个月；CLAP 同时用于训练 reward 和评测指标构成循环；遗漏 V2A-DPO/Kling-Foley/VAFlow/TARO 等关键 baseline；8人同实验室 user study 独立性不足。
  • Full review: Claude Code 全文七公理审稿

Video-to-audio (V2A) generation faces significant challenges in achieving precise temporal synchronization and high perceptual quality due to the complex, ambiguous relationship between visual and auditory cues. Existing methods typically compress video inputs into single feature representations, leading to significant loss of temporal dynamics and fine-grained visual information. These approaches also rely on reconstruction-based training objectives that poorly correlate with human perceptual judgments of audio quality and appropriateness. We propose HarmoniDPO, a novel framework that integrates preference-based optimization into diffusion-based V2A generation to address these limitations. (1) Our approach leverages a dual video representation: combining global context with frame-wise features to preserve temporal dynamics and semantic detail. (2) Inspired by reinforcement learning from human feedback (RLHF), HarmoniDPO employs online Direct Preference Optimization (online-DPO) to fine-tune a diffusion-based V2A model from preference judgments, enhancing perceptual quality and alignment. (3) Additionally, we introduce Dual-scale Diffusion Search (DDS), a test time scaling algorithm that adaptively optimizes output fidelity during inference. Experiments demonstrate that HarmoniDPO outperforms state-of-the-art methods in audio-video synchronization and subjective audio quality, offering a robust solution for generating realistic, human-preferred audio from video.

16. Rethinking Language Model-Based Generative Speech Enhancement in the Latent Space of a Neural Audio Codec

Authors: Yihui Fu, Zhengyang Li, Tim Fingscheidt

Categories: eess.AS | Accepted at IWAENC 2026 Score: 4.60/10 (Obj:5 Id:5 Ind:4 Comp:5 Eff:5 Nov:3 Rep:8)

  • Strength: 在统一实验框架下对比六种 LM-based 生成式 SE 范式，发现 continuous NAR (CNAR) 最优，并提出辅助损失微调策略在所有范式上一致提升 DNSMOS/NISQA/PESQ/POLQA。
  • Weakness: 核心发现（continuous > discrete, NAR > AR）已被 Kammoun et al. (2510.26299, 2025年10月) 报告，”first to compare” 声称不成立；辅助损失微调将 differentiable PESQ/STOI 用于训练而在 PESQ/ESTOI 上评测，存在循环论证风险；未与 URGENT 2025 Challenge leaderboard SOTA 对比。
  • Full review: Claude Code 全文七公理审稿

Language model (LM)-based speech enhancement (SE) has recently emerged rapidly using latent space features of neural audio codecs (NACs). In this paper, first, we present a unified framework covering six popular LM-based generative SE modeling paradigms based on discrete/continuous latent NAC features: discrete or continuous autoregressive (D/CAR) SE, discrete or continuous non-autoregressive (D/CNAR) SE, discrete diffusion (DDiff) SE, and continuous flow matching (CFM) SE. Second, we are the first to compare their performance in a unified experimental setup and synopsis with diverse intrusive and non-intrusive metrics, enabling a fair and comprehensive evaluation. Third, we propose a fine-tuning strategy with auxiliary losses on reconstructed speech to improve both intrusive and non-intrusive metrics. Trained and evaluated on URGENT 2025 Speech Enhancement Challenge data splits, all continuous-domain paradigms excel their discrete-domain counterparts. The overall best approach turns out to be CNAR. We further show that our proposed auxiliary loss fine-tuning strategy helps to improve DNSMOS, NISQA, PESQ, and POLQA consistently in all six paradigms.

17. CookVoice: Unified Framework for Style Controllable Multi-Modal Human Voice Generation

Authors: Haowei Lou, Hye-Young Paik, Dai Jia, Kai Li, Lina Yao

Categories: cs.SD, cs.LG Score: 3.68/10 (Obj:5 Id:2 Ind:5 Comp:4 Eff:5 Nov:2 Rep:5)

  • Strength: 43.51M参数统一框架支持6种语音生成任务（TTS/SVS/风格控制/拟声/转换/编辑），4步ODE推理，在部分指标上接近F5-TTS和Vevo2等大规模基线。
  • Weakness: 核心创新与Vevo2（2025.08）和UniVoice（2026.06）高度重叠，未隔离任何组件的因果贡献，且在TTS的WER和SECS、SVS的WER等核心指标上均未超越已有基线，关键统一基线UniVoice被遗漏。
  • Full review: Claude Code 全文七公理审稿

Human voice generation has made rapid progress in speech generation, singing voice generation, voice cloning, and voice editing. However, most existing systems are designed for specific tasks and often rely on task-dependent architectures, control signals, or autoregressive decoding, limiting fine-grained controllability and inference efficiency. In this paper, we propose CookVoice, a unified framework for multimodal, multi-style, and multi-task human voice generation. CookVoice decomposes the human voice into three key factors: content, prosody, and style, enabling both speech and singing voice generation within a unified model. To achieve precise and flexible controllability, we design a flexible alignment strategy that maps text, style, and prosody control signals onto the frame-level of spectrogram. This design allows CookVoice to support a wide range of tasks, including text-to-speech, text-to-singing voice, style-controllable generation, voice mimicry, voice conversion, and voice editing. Experimental results show that CookVoice achieves generation quality comparable to existing Text-to-Speech and text-to-singing voice baselines, while providing stronger style and prosody controllability. Moreover, CookVoice achieves comparable performance to large-scale baselines with only 43.51 million parameters and efficient inference using as few as 4 ODE steps, making it a practical solution for real-world human voice generation applications. Demo page is available at https://haoweilou.github.io/CookVoice/.