每日 arXiv

音频 · 语音 · 音乐 · 声学

每日新爬取的 arXiv 论文，按评分排序。中文摘要帮你快速判断是否值得深读。

2026-08-02
日期2026-08-02
已评分—
均分—
最高—

Daily Papers — 2026-08-02

7 papers on audio, speech, music, and acoustics.

1. FATE: Frame-Level Audio-Visual Temporal Embedding

Authors: Kaisi Guan, Bingzi Zhang, Xihua Wang, Ying Ba, Xin Cheng et al.

Categories: cs.MM Score: 8.21/10 (Obj:9 Id:8 Ind:9 Comp:9 Eff:7 Nov:8 Rep:9)

  • Strength: FATE 通过保留帧级嵌入并在物理时间轴上对齐，以极简设计（去掉 PE-AV 池化头+帧级对角相似度+联合语义-时序对比损失）将音视频检索 R@3 提升 17+ points，零样本事件定位 48.3% 匹配有监督方法 47.8%，推理 15× 快于 Synchformer。
  • Weakness: 生成评估的绝对 sample-level Spearman 相关仅 17.24%（作者承认”未闭合”），model-ranking ρ 仅基于 5 个生成模型，且未与 CVPR 2025 的 Open-Vocabulary AVEL 等 2 个潜在相关基线比较。
  • Full review: Claude Code 全文七公理审稿

When a dog opens its mouth and barks, humans naturally recognize what the sound is and when it occurs. Building audio-visual models with this same ability requires representations that capture both semantic and temporal alignment. Current approaches fall short on one side or the other: embedding models match semantic but lose temporal information; synchronization models capture temporal offsets but lack semantic understanding. To bridge this gap, we propose FATE, Frame-level Audio-visual Temporal Embedding. Unlike prior embedding models that pool each modality into a single embedding and discard temporal information, FATE retains frame-level sequences, aligns them on the physical timeline, and computes similarity over strictly aligned frame pairs. Unlike synchronization models that output only an offset prediction, FATE encodes synchronization in a reusable embedding space, trained with a joint objective combining cross-video semantic and within-video temporal contrastive learning to capture both what sounds and when it occurs. Across three tasks, FATE surpasses the strongest baseline on temporal and semantic retrieval by a large margin, matches fully supervised methods on event localization in a zero-shot setting, and achieves the best correlation with human judgments as a generation evaluation metric. The source code can be found at \texttt{https://github.com/guankaisi/FATE}.

2. Interpretable MEG Decoding of Perceived Speech: Cortical Sources and the Stimulus Features That Drive Retrieval

Authors: Ilia Semenkov, Daria Kleeva, Ivan Dakhtin, Zarina Maksudova, Alex Ossadtchi

Categories: cs.LG, cs.SD, q-bio.NC Score: 8.00/10 (Obj:9 Id:8 Ind:9 Comp:8 Eff:7 Nov:8 Rep:8)

  • Strength: 用球谐函数3D空间注意力+时空分支前端（K=25, ~487K参数，约20×少于Défossez et al.）在MEG-MASC上达到39.75% Top-1检索精度，权重可映射到皮层源空间并恢复语音感知网络，配对MEG occlusion识别出19个刺激特征中15个显著贡献（最大为静默、响度、元音、声学起始），并发现随机词表段的负向occlusion效应。
  • Weakness: 与Défossez et al.的直接比较不完全公平（预处理、评测协议、候选数均不同），绝对精度未超过前作（39.75% vs 41.3%），且仅单数据集27被试验证，泛化性未得到外部确认。
  • Full review: Claude Code 全文七公理审稿

Short segments of perceived speech can be retrieved from non-invasive magnetoencephalographic (MEG) recordings by deep networks trained with a CLIP-style objective against wav2vec 2.0 audio embeddings. Yet their weights do not map onto electrophysiological quantities, and it remains unclear which speech properties drive retrieval. We build on a high-performing MEG-to-audio retrieval architecture but redesign both its front end and decoder. Its spatial attention operates on a flattened sensor layout; we replace it with spherical harmonics defined on the three-dimensional MEG helmet geometry. We reduce the subject-specific representation from 270 to 25 branches, add a temporal filter to each branch to match it to a neuronal source in space and time, and make the convolutional decoder shallower. Ocular and cardiac components are removed before training to reduce the risk of stimulus-locked shortcuts. On MEG-MASC, the model reaches 39.75 +/- 0.34% Top-1 accuracy among 1005 candidates across six trained solutions, with about 20 times fewer decoder parameters. Its weights map to source space, recovering generators consistent with the speech-perception network, while left-lateralized branches carry higher-frequency rhythmic components not evident on the right. Paired MEG occlusion shows that 15 of 19 stimulus features contribute, with the largest effects for silence, sound intensity, vowels, and acoustic onsets. Random word lists behave oppositely: substituting narrative MEG into them improves retrieval, indicating that activity without narrative structure carries less recoverable information than activity during coherent speech. The wav2vec target can be reduced to about twelve learned feature dimensions without loss of accuracy, whereas strong temporal compression causes a clear loss. Together, source mapping and input interventions reveal what drives retrieval.

3. InteracVid: Building a Real Interactive Audio-Visual Response Dataset from Live-Chat Videos

Authors: Chi Zhang, Haoyang Shi, Yueyi Liu, Zhaokun Yan, Yishu Yin et al.

Categories: cs.CV Score: 7.16/10 (Obj:9 Id:6 Ind:8 Comp:6 Eff:7 Nov:8 Rep:6)

  • Strength: 构建了首个大规模交互式音视频响应数据集（454K 三元组，59K 直播视频），微调在 3 个 VLM backbone 上一致提升 generator OVRL +0.84–1.62，人类评估与自动 judge Pearson r 达 0.949–0.979。
  • Weakness: 缺少在非交互式 AV 数据上训练的对照实验以隔离”交互结构”的因果贡献，且数据构建与评测分别依赖闭源 Seed-1.6 和 Gemini-3.1-Pro，pipeline 组件无消融。
  • Full review: Claude Code 全文七公理审稿

Large language models have made text the default medium for human–AI interaction, buttext alone cannot express the full range of responses required by multimodal assistants,avatars, and embodied agents. While recent audio-video generative models can synthesizehigh-fidelity synchronized content, existing supervision is largely \emph{descriptive}:models are trained to render captions rather than to produce audio-visual responsescaused by external user interactions. We introduce \textbf{InteracVid}, \emph{the firstopen-source large-scale dataset that addresses this missing supervision}, so that everysample couples a preceding audio-visual context and an external stimulus with the realinteractive response that follows. We design a metadata-aware pipeline that extractsinteractive clips from long, noisy livestreams, yielding over \textbf{454K}context-query-response triplets from more than \textbf{59K} livestream videos andspanning conversation-centered, object-centric, procedural, embodied, and screen-basedscenarios. A ten-rater human study confirms that the extracted interactions are causal,natural, and temporally complete for both genuine and reconstructed queries. On aheld-out benchmark of \textbf{100} genuine live-chat queries, fine-tuning on InteracVidimproves both interaction planning and audio-video response generation, and anindependent human evaluation reproduces the system ranking and the conclusions obtainedwith our automatic judge. These results highlight interaction-structured data as acritical foundation for interactive multimodal generation.

4. Latent Softmax for Data-Efficient Phoneme-Based Multilingual ASR Across Tonal and Non-Tonal Languages

Authors: Saierdaer Yusuyin, Nanling Jiang, Hao Huang, Zhijian Ou

Categories: eess.AS, cs.LG Score: 6.30/10 (Obj:8 Id:5 Ind:8 Comp:8 Eff:6 Nov:7 Rep:5)

  • Strength: Latent Softmax 仅修改 CTC 输出层概率计算（子类边缘化），在 Mandarin-English 多语言训练中实现 S2P PER 相对下降 8.4%-17.5%，方法极简且有两套等价梯度推导。
  • Weakness: 仅验证 1 个声调语言（Mandarin），缺少 collapsed-softmax baseline 和与同类方法（JoinAP、Universal allophone）的实验对比，ASRU2019 上 LLM-P2G 出现负向结果未充分讨论，代码未开源。
  • Full review: Claude Code 全文七公理审稿

Phoneme-based multilingual automatic speech recognition (ASR) can share acoustic evidence across languages more directly than language-specific subword modeling. When tonal and non-tonal languages are jointly trained, however, their supervision granularity does not match: tonal languages annotate tone-marked vowels, whereas non-tonal languages typically provide only base-vowel labels. A standard softmax either treats the two as unrelated classes, weakening cross-lingual sharing, or collapses tones, losing distinctions required by tonal languages. We propose Latent Softmax, a connectionist temporal classification (CTC)-compatible output layer that models tone-marked vowels as subclasses and base vowels as major classes, while consonants and the CTC blank remain singleton labels. When only a base-vowel major-class label is observed, the tone-marked vowel subclass is treated as latent and marginalized out. Multilingual experiments on AISHELL-1 Mandarin and LibriSpeech English show that Latent Softmax reduces speech-to-phoneme (S2P) phoneme error rates over a standard softmax multilingual baseline by 8.4% on AISHELL-1, 17.5% on LibriSpeech test-clean, and 12.6% on test-other. The improved speech-to-phoneme encoders also yield consistent word error rate gains for both large-language-model phoneme-to-grapheme conversion and projector-based interfaces. After code-switching adaptation, Latent Softmax further reduces projector-based mixed error rate by 2.6% on ASRU2019 and 9.5% on CS-Dialogue datasets.

5. JoyAI-Talker: Full-Duplex Speech Interactive Large Model Built for Empathetic Voice Agents

Authors: Yinhao Bai, Jinming Chen, Yafeng Chen, Wei Deng, Boya Dong et al.

Categories: cs.SD Score: 5.95/10 (Obj:8 Id:5 Ind:5 Comp:5 Eff:8 Nov:5 Rep:3)

  • Strength: Full-Duplex-Bench v1.5 全面 SOTA（interruption response 0.88 vs GPT-4o 0.78/Gemini 0.77，background false-trigger 0.10 vs Gemini 0.28），T2T 在 MATH 达 94.62% 证明 cognitive degradation 得到有效缓解，系统在绝对可用水平上实现全双工共情语音交互。
  • Weakness: 纯系统工程无方法论消融（无 joint-training-stage ablation、无 PAER-without ablation、无 Joy-Duplex-without ablation），全部模块/数据/judge 闭源且无 code release，每个组件均有明确先例（Thinker-Talker from Qwen3-Omni, JoyVoice Talker, SoulX-Duplug/FastTurn duplex。
  • Full review: Claude Code 全文七公理审稿

We present JoyAI-Talker, a full-duplex speech dialogue system that delivers robust foundation model capabilities while empowering empathetic interaction and voice agent intelligence. JoyAI-Talker adopts a modular Thinker-Talker architecture and further implements a unified speech-text joint training pipeline to mitigate the common “cognitive degradation” bottleneck, thereby largely preserving the model’s core textual reasoning, STEM, and logical capabilities while extending them to speech-based interaction. For expressive speech synthesis, the Talker module employs a text-controllable generation paradigm that enables natural-language instructions to flexibly control vocal attributes and localized paralinguistic events, such as laughter and sighs, supporting more expressive and fine-grained speech responses. To enhance conversational empathy, we introduce the Persona-Adaptive Empathetic Response (PAER) framework. PAER employs a hierarchical cognitive pipeline to extract non-verbal speaker cues, such as gender, age, and emotional state, from raw input audio, incorporate them into the Thinker’s CoT reasoning, and generate context-adaptive responses that align semantically appropriate text with fine-grained control over utterance-level expressiveness and localized paralinguistic events, including sighs, speaking rate, and volume. We further integrate Joy-Duplex, a state-driven, plug-and-play full-duplex framework that functions as an efficient gating engine for real-time turn control. Extensive evaluations show that JoyAI-Talker achieves highly competitive performance on foundational T2T and S2T benchmarks. In full-duplex evaluation, the system reaches a high response rate of 0.88 under user interruptions while maintaining an extremely low false-trigger rate under background speech, demonstrating its readiness for fluid and natural speech dialogue.

6. Separate-and-Detect: Unified Drum Transcription and Stem Generation via Latent Diffusion

Authors: Wei-Han Hsu, Chih-Cheng Chang, Bo-Yu Chen, Li Su, Yi-Hsuan Yang

Categories: cs.SD | 8 pages, 3 figures, 4 tables. Accepted at the 27th International Society for Music Information Retrieval Conference (ISMIR 2026) Score: 5.74/10 (Obj:8 Id:5 Ind:6 Comp:5 Eff:6 Nov:5 Rep:9)

  • Strength: 在 MDB/ENST 上 overall F1 显著优于 LarsNet（p<0.01），kick F1 达 0.931、snare 0.760 超越 ADTOF，同时额外提供五条可编辑 drum stems（ADTOF 无法提供），代码与 demo 开源。
  • Weakness: “Unified” 名实不符——toms/hi-hats/cymbals 的 F1（0.268/0.650/0.481）远低于 ADTOF，仅 kick/snare 胜出；方法为 Demucs+MSG-LD+madmom 的 sequential 组合，缺最简 baseline 隔离 latent diffusion 分离器本身的必要性，且推理 25s/10s 样本限制实用性。
  • Full review: Claude Code 全文七公理审稿

Automatic Drum Transcription (ADT) is commonly formulated as a direct mapping from a music mixture to symbolic drum events. While effective for transcription, this formulation discards the acoustic stems that are useful for editing, remixing, and production. We revisit an alternative separate-and-detect formulation, where a drum source separation front end first produces five editable drum stems, and a fixed onset detector then converts each stem into symbolic events. The separator is built on a five-stem latent diffusion model that jointly generates kick, snare, toms, hi-hats, and cymbals in a compact VAE latent space. We further study two training-only auxiliary branches–an onset branch (OB) and a timbre branch (TB)–which shape the separator during learning but are discarded at inference. Trained on synthetic drum multitracks and evaluated on MDB Drums and ENST-Drums, the proposed pipeline consistently improves over a strong U-Net-based drum separation baseline in overall transcription F1. It also outperforms a representative end-to-end ADT system on kick and snare F1 under our evaluation protocol, while additionally providing separated audio stems. The ablation results show that OB gives the most stable transcription gains, whereas TB changes the trade-off between reconstruction, acoustic stem quality, and onset detection. These results suggest that generative drum demixing can serve not only as a source separation model, but also as a practical front end for interpretable drum transcription.

7. Beyond One-Size-Fits-All: Personalized and Culturally Adaptive Emotional TTS via Interactive Optimization of Individual Emotion Perception Spaces

Authors: Wangzixi Zhou, Bagus Tris Atmaja, Sakriani Sakti

Categories: eess.AS | Accepted for publication at INTERSPEECH 2026 Score: 4.70/10 (Obj:6 Id:5 Ind:3 Comp:5 Eff:5 Nov:4 Rep:6)

  • Strength: 提出将情感 TTS 个性化压缩为 2D A-V 坐标的 IGA 搜索问题，30 人跨 3 文化实验显示个人化偏好率 76%、文化适配偏好率 65-70%，证明情感感知的个体/文化差异在 TTS 中确实存在且可操作化适配。
  • Weakness: 训练标注与客观评测均依赖同一 SER 模型构成循环论证，Table 1 的 emotion controller 提升同时改变多个变量但无 ablation 拆解，IGA 在 2D 空间的必要性未对比简单搜索方法验证，仅 3 个亚洲文化组无法支撑”culturally adaptive”的广义 claim。
  • Full review: Claude Code 全文七公理审稿

The rise of conversational AI has increased interest in emotional Text-to-Speech (TTS). Most systems rely on discrete emotion labels, which fail to capture the nuanced nature of human affect. Recent models employ dimensional representations such as Russell’s arousal-valence (A-V) model, offering finer control. However, emotional perception varies across individuals and cultures, which may cause mismatches between modeled and perceived emotions. We propose a personalized and culturally adaptive emotional TTS framework that performs interactive optimization of individualized A-V perception spaces using an Interactive Genetic Algorithm. By adapting emotion representations to each listener, the system produces speech with more perceptually aligned emotional expression than models using averaged A-V values. Evaluations with Japanese, Chinese, and Indonesian participants highlight the importance of personalization and cultural adaptation for moving beyond one-size-fits-all emotional TTS.