每日 arXiv

音频 · 语音 · 音乐 · 声学

每日新爬取的 arXiv 论文，按评分排序。中文摘要帮你快速判断是否值得深读。

2026-07-10
日期2026-07-10
已评分—
均分—
最高—

Daily Papers — 2026-07-10

8 papers on audio, speech, music, and acoustics.

1. ReGen: Hierarchical Multi-Prompt Representation Generation for Efficient Waveform Diffusion Models

Authors: Sang-Hoon Lee, Ha-Yeong Choi

Categories: cs.SD, cs.AI, eess.AS, eess.SP | Accepted to ICML 2026 Score: 6.40/10 (Obj:8 Id:6 Ind:8 Comp:5 Eff:8 Nov:5 Rep:8)

  • Strength: ReGenVAE 在 12.5 Hz/32 dim 极低比特率下实现 WER 2.88、SPK-SIM 0.87、UTMOS 4.096 的重建质量，显著优于 Mimi/WavTokenizer 等基线；ReGenVoice 以 0.5B 参数、4 GPU 1 天训练达到 RTF=0.08 和 WER 1.62 的 TTS 性能。
  • Weakness: 核心方法是对已有技术（REPA-H、masked-infilling、GED 排斥项、对抗后训练）的工程组合，缺乏对层级顺序的理论解释，且消融中 GFM 在 WER 上未见改善；未验证”调低 REPA 权重”这一最简替代方案。
  • Full review: Claude Code 全文七公理审稿

Representation alignment (REPA) has been investigated to accelerate diffusion training, but we observe that regularizing intermediate representations in diffusion Transformers (DiT) may implicitly entangle latents and limit generative capacity. To address this issue, we propose ReGen, a hierarchical multi-prompt representation generation framework that jointly estimates multiple vector fields for both representations and data within a single diffusion model. We further introduce generalized flow matching (GFM) to improve the generalization of conditional flow matching (CFM). We validate ReGen on single-stage waveform diffusion models including neural audio codec and Wave-VAE. ReGen significantly improves waveform generation quality from highly compressed latent representations at 12.5 Hz. We also present ReGenVoice, a latent diffusion model (LDM)-based text-to-speech model that achieves strong speech intelligibility (WER) and speaker similarity (SIM) with a small dataset. Moreover, operating the LDM at 6.25 Hz with rich semantic and acoustic latent representation enables efficient training and sampling, requiring only 1 day of training on 4 GPUs and fast inference with an RTF of 0.08. Audio samples are available at https://regenvoice.github.io/demo/.

2. Tokenizer Transplantation: Mitigating Autoregressive Collapse in Edge-Efficient Bengali ASR

Authors: Sanjid Hasan, Md. Abdur Rahman

Categories: cs.CL | 5 pages, 2 figures. Accepted as a poster at the MusIML Workshop, ICML 2026 Score: 6.05/10 (Obj:8 Id:5 Ind:6 Comp:8 Eff:5 Nov:5 Rep:8)

  • Strength: 通过 tokenizer 移植将 token fertility 从 9.16 降至 1.30，在 61.5M 参数下实现 RTF 0.0053（比 Whisper-Medium 快 3.5x），CER 10.79% 为所有对比模型最低，代码和数据完全开源。
  • Weakness: WER 21.54% 实际高于 Fine-Tuned Faster Whisper Medium（21.28%），无任何消融实验隔离三阶段 pipeline 各组件贡献，核心 WER 结果仅在自建数据集上验证，词表迁移方法在 LLM 领域已有充分先例（WECHSEL/FOCUS/Trans-Tokenization）。
  • Full review: Claude Code 全文七公理审稿

Lightweight speech recognition models are critical for edge deployment, yet highly optimized architectures like Moonshine often fail on morphologically rich, non-Latin languages such as Bengali. This study identifies the root cause of this failure as the model’s English-centric byte-level tokenizer, which fragments Bengali words into high-fertility byte chains and triggers catastrophic autoregressive collapse during inference. To resolve this, a novel vocabulary transplantation pipeline is proposed to replace the decoder vocabulary with the native-script BanglaBERT WordPiece vocabulary and resize the corresponding token embedding matrix. Experimental results demonstrate a reduction in token fertility from 9.16 to 1.30. By decreasing autoregressive sequence length by 85.8%, decoding instability is entirely mitigated. When evaluated on the 882-hour Lipi-Ghor dataset, the modified architecture achieves a competitive 21.54% Word Error Rate (WER) and a Real-Time Factor (RTF) of 0.0053. Ultimately, this research provides a scalable, reproducible blueprint for cross-script adaptation of compact ASR models without the need for resource-intensive pre-training.

3. Beyond Time Shifts: Adapting Omni-LLM as a Reference-Free Evaluator for Generative Audio-Visual Models

Authors: Yijie Qian, Juncheng Wang, Chao Xu, Huihan Wang, Yuxiang Feng et al.

Categories: cs.CV | Accepted to ECCV 2026 Score: 5.95/10 (Obj:8 Id:5 Ind:5 Comp:5 Eff:5 Nov:4 Rep:6)

  • Strength: 构建了首个真实生成失败AV同步数据集SynthSync（2,267 anchors, 306K pairwise annotations），在BoN跨指标泛化中4个独立metric中3个达到最优off-diagonal性能，确认reward信号的通用性。
  • Weakness: 所有定量评测仅在SynthSync test split上完成，无独立外部验证集；R-GRPO相对BTL-only baseline增量仅+0.9% NDCG，引入6项composite reward的工程复杂度与收益不成比例；Kendall’s τ=0.49、Pair-Acc=72.38%表明metric仍有约28%的pairwise判断错误。
  • Full review: Claude Code 全文七公理审稿

As audio-visual generative models evolve into world simulators, cross-modal synchronization stands as a critical proxy for assessing the consistency of world dynamics and causality in generated content. However, existing evaluation metrics presume structural correctness, reducing synchronization to mere temporal alignment. Consequently, they fail on generative outputs, especially when exhibiting structural hallucinations and asymmetric cross-modal relations, which currently \textbf{mandate expert human annotation to assess synchronization.} This dependency introduces a critical paradox: \emph{human evaluators rely on relative, reference-dependent comparisons, whereas automated metrics require reference-free, absolute scalars.} We resolve this paradox by proposing a framework that distills relative human perception into a continuous, globally consistent metric. First, we introduce SynthSync, a dataset of generative failures ranked via pairwise human annotations. Second, we adapt the Omni-LLM equipped with a continuous latent projection to translate relative human rankings into continuous absolute values. Third, we propose Real-Valued Group Relative Policy Optimization ($\mathbb{R}$-GRPO) to internalize the global causal structure of synchronization via listwise score distributions. Empirically, our metric achieves state-of-the-art human preference alignment. We leverage this estimator to establish a standardized benchmark, advancing AV-Gen assessment from low-level signal correlation to visually grounded causality.

4. Phone Segmentation and Recognition through Phonological Activation Mapping

Authors: Shikhar Bharadwaj, Kwanghee Choi, Stephen McIntosh, Chin-Jou Li, Eunjung Yeo et al.

Categories: eess.AS, cs.AI, cs.CL, cs.LG, cs.SD | Code will be released after acceptance Score: 5.90/10 (Obj:8 Id:6 Ind:8 Comp:5 Eff:6 Nov:5 Rep:9)

  • Strength: 仅需 <1 分钟标注数据、47K 参数即可在 95 种语言的 out-of-domain 分割任务上达到 R-value 75.1（显著优于 316M 参数的 FCE 的 63.0），并实现零样本识别训练中未见过的 phone。
  • Weakness: In-domain phone recognition 绝对性能严重不足（TIMIT PFER 22.9 vs CTC 7.2），分割头依赖 7 种信号 ensemble（含经典 mel spectrogram 信号），且核心表示 SPAM 和 phonological vectors 均来自同一研究组的先前工作，真正的方法增量有限。
  • Full review: Claude Code 全文七公理审稿

Phone segmentation and recognition are inherently related tasks, yet modern approaches typically model them separately. We argue that phonetic structure is already latent in the representations of self-supervised speech models (S3Ms), and one only needs to steer them to solve both tasks. We leverage S3M-based Phonological Activation Mapping (SPAM), which maps each S3M representation frame to a vector of phonological feature activations, such as voicing and nasality. On top of SPAM, we introduce two simple but effective lightweight, gradient-descent-free prediction heads: a recognition head and a segmentation head. Our method requires less than a minute of phonetic transcriptions, and generalizes to unseen phones during training. Across a diverse range of datasets, our approach attains strong segmentation and recognition performance.

5. Technical Report for MERL’s Real-TSE Challenge Submission

Authors: Dominik Klement, Yoshiki Masuyama, Christoph Boeddeker, Kohei Saijo, Julius Richter et al.

Categories: eess.AS Score: 5.30/10 (Obj:6 Id:5 Ind:5 Comp:5 Eff:8 Nov:3 Rep:5)

  • Strength: 通过精心设计的四阶段数据管线（合成预训练→噪声模拟→远场适应→真实混合训练）在 Real-TSE Challenge Track 2 获得第一名，TER 从 baseline 的 0.53 降至 0.37，并揭示了 DNSMOS/speaker similarity 可被对抗攻击推至极端值而不影响 TER/F1。
  • Weakness: 无方法新颖性（所有组件均为已有工具的直接组合），关键变量未隔离消融（同时改变模型规模、数据、损失函数），评测指标与训练损失重叠（DNSMOS 和 speaker similarity 既用于训练又用于评测）。
  • Full review: Claude Code 全文七公理审稿

Target speech extraction (TSE) has largely been dominated by neural network-based approaches trained and evaluated on synthetic fully overlapped data. The Real-TSE Challenge aims to advance performance on real-world far-field noisy and reverberant recordings. This technical report describes MERL’s submission to the Real-TSE Challenge. Rather than proposing a novel model architecture, we built upon the baseline model and focused primarily on data preparation and cleaning. Our system was trained in four stages, beginning with pre-training on fully overlapped mixtures and simulated multi-talker conversations with noise and reverberation applied to both the mixture and the enrollment utterances. We then adapted the model to real-world conditions using noisy far-field recordings with pseudo-targets derived from processed close-talk microphone signals. Our submission achieved first place in the second track, demonstrating the critical importance of high-quality data preparation. Furthermore, we observed that DNSMOS and speaker similarity are susceptible to over-optimization, motivating an investigation of their robustness using adversarial attacks. The results show that both metrics can be driven to extreme values without degrading the token error rate or the VAD-based F1 score.

6. Event-Based Token Sequences for Audio-Conditioned Music-Game Level Modeling

Authors: Ke Zhang, Chu-Hsuan Hsueh, Kokolo Ikeda

Categories: cs.SD, cs.MM | Camera-ready version, published at ICMR 2026 Score: 5.00/10 (Obj:8 Id:5 Ind:6 Comp:7 Eff:4 Nov:4 Rep:5)

  • Strength: 提出将音乐游戏关卡生成从 frame-level 重构为 event-based token-level seq2seq 问题，在 maimai 数据集上 Event F1 达 0.527，较 frame-based baseline DDC（0.254）和 GeneLive!（0.298）有显著提升，并引入 ACS 指标量化音频贡献。
  • Weakness: 遗漏最接近的同期工作 Yi et al. 2023（beat-aligned spectrogram-to-sequence for rhythm game charts）和 TaikoNation (2021) 未作比较，绝对 Event F1 仅 0.527 远未达可部署水平，且未隔离”表示方式”与”模型架构”各自的贡献。
  • Full review: Claude Code 全文七公理审稿

Procedural generation of music game levels is an exciting yet challenging problem, as levels must translate musical structure into interactive sequences of timed gameplay events. Most existing approaches formulate this task by frame-based representations, dividing audio into uniform time grids and predicting events at each frame. This makes gameplay events implicit across many frames. As a result, it is hard to describe event-level timing relations and longer-range structure found in human-authored levels. We use procedural generation as a practical setting to study how musical cues map to interactive event sequences. Inspired by event-based symbolic music modeling, we propose a token-level sequence formulation that casts level generation as a multimodal sequence-to-sequence problem. Conditioned on an audio excerpt and level metadata, the model generates a token sequence alternating gameplay-event and beat-shift tokens. This explicitly represents actions and their relative timing in beat space. Based on this formulation, we build a Transformer model. It outperforms representative frame-level baselines under event-level evaluation. It also enables systematic analysis of how audio supports rhythm-aligned event prediction beyond metadata conditioning.

7. Optimal Transport-based Semantic Alignment for LLM-based Audio-Visual Speech Recognition

Authors: Xugang Lu, Peng Shen, Yu Tsao, Hisashi Kawai

Categories: cs.SD, eess.AS Score: 4.84/10 (Obj:7 Id:5 Ind:6 Comp:5 Eff:6 Nov:4 Rep:3)

  • Strength: OT 耦合矩阵作为对比学习软标签的机制在 LRS3-TED clean 条件下取得 WER 0.71%，相对最强基线 MMS-LLaMA 提升 25%，且提供了融合策略消融和超参数敏感性分析。
  • Weakness: 核心方法本质上是作者前序 OT-ASR 工作 [18,19] 向 LLM-AVSR 的增量扩展，缺少与同期 LLM-AVSR 模态对齐工作 AVUR-LLM（2603.03811）和 Omni-AVSR（2511.07253）的比较，且未开源代码、关键超参数不完整，复现困难。
  • Full review: Claude Code 全文七公理审稿

Large language model (LLM)-based audio-visual speech recognition (LLM-AVSR) has recently demonstrated strong robustness in adverse acoustic environments by leveraging complementary audio and visual information. Existing approaches typically employ independently pretrained acoustic and visual encoders, whose outputs are projected and fused as soft prompts to condition an LLM for speech recognition. However, most methods perform multimodal fusion without explicitly addressing the representational discrepancy between audio, visual and text modalities, potentially limiting the effectiveness of cross-modal integration. In this paper, we propose an optimal transport (OT)-based semantic alignment framework for LLM-AVSR. The proposed method explicitly bridges the modality gap by aligning the acoustic and visual representations with reference to the linguistic embedding space of the LLM before multimodal fusion. Specifically, OT is used to estimate probabilistic coupling matrices that characterize structured correspondences between modality-specific features and linguistic embeddings. The resulting OT couplings are further utilized as soft pseudo-labels to supervise contrastive learning, encouraging the extraction of semantically coherent and cross-modal consistent audio-visual representations. By anchoring multimodal features to the linguistic space of the LLM, the proposed framework facilitates more effective multimodal fusion and decoding. We implement the proposed framework using a Whisper-based acoustic encoder, an AV-HuBERT-based visual encoder, and a LLaMA3.2-3B decoder. Experiments conducted on the LRS3-TED benchmark demonstrate consistent improvements over strong baselines and achieve state-of-the-art performance under both clean and noisy evaluation conditions across a wide range of signal-to-noise ratios (SNRs).

8. Wan-Dancer: A Hierarchical Framework for Minute-scale Coherent Music-to-Dance Generation

Authors: Mingyang Huang, Peng Zhang, Li Hu, Guangyuan Wang, Bang Zhang

Categories: cs.CV, cs.SD | 17 pages, 13 figures, project: https://github.com/Wan-Video/Wan-Dancer Score: 4.70/10 (Obj:5 Id:3 Ind:5 Comp:5 Eff:5 Nov:5 Rep:5)

  • Strength: 首个实现 minute-scale（>60s）720p/30fps music-to-dance video 生成的端到端系统，通过 global keyframe planning + local temporal refinement 的分层推理，在 5 种舞蹈风格上实现 user study 评分显著优于 X-Dancer 和 MusicInfuser。
  • Weakness: 仅与 2 个非长视频 baseline 比较，未与任何专门的长视频生成方法（FreeNoise/FramePack/TempoMaster）或同期舞蹈生成方法（MACE-Dance/ChoreoMuse）对比；ablation 全部为定性 figure 展示无定量指标；多模态输入不对等（A+T+I vs A+I / A+T）使比较不公平；训练数据为 200h proprietary dataset 不可复现。
  • Full review: Claude Code 全文七公理审稿

Generating long-duration, high-definition, and rhythmically synchronized dance videos directly from music remains a significant challenge, primarily due to the temporal constraints of current diffusion models, which typically fail beyond 20 seconds. Existing approaches, whether they rely on intermediate 3D skeletons or on end-to-end video synthesis, suffer from temporal drift, identity inconsistency, and repetitive motion patterns when extended to longer horizons. To address these limitations, we propose a novel hierarchical framework for minute-scale coherent music-to-dance generation. Our method decouples the process into global keyframe planning and local temporal refinement, leveraging full-track musical context to ensure long-range coherence. Key innovations include dynamic frame rate adaptation via time-mapped RoPE embeddings for precise alignment, an optical-flow-based loss function to enhance motion continuity, and motion-speed control to preserve high-fidelity details during rapid movements. Extensive experiments demonstrate that our framework surpasses the conventional duration barrier, generating stable, 720p/30fps videos exceeding one minute with superior temporal stability. Furthermore, the model exhibits robust versatility across five distinct dance genres, conditioned on both audio and textual prompts, establishing a new state-of-the-art in coherent, long-form dance video synthesis.