每日 arXiv

音频 · 语音 · 音乐 · 声学

每日新爬取的 arXiv 论文，按评分排序。中文摘要帮你快速判断是否值得深读。

2026-08-10
日期2026-08-10
已评分—
均分—
最高—

Daily Papers — 2026-08-10

22 papers on audio, speech, music, and acoustics.

1. MADBench: A Benchmark for Modality-Aware Audio Deepfake Detection

Authors: Yanqiu Li, Yang Xiao, Jisheng Bai, Bin Chen, Hong Jia et al.

Categories: cs.SD, cs.AI | 11 pages, 1 figure Score: 6.63/10 (Obj:8 Id:8 Ind:6 Comp:8 Eff:7 Nov:5 Rep:5)

  • Strength: 首个在真实视频设置下独立操控语音和环境音频的音视频 deepfake 基准测试，揭示了一个清晰的迁移层级（预训练检测器失败 AUC~0.52，冻结编码器成功 AUC~0.95）以及非对称跨组件干扰（虚假环境音频降低语音检测 GS 高达 0.138，但反之不然）。
  • Weakness: 核心组件感知概念（独立语音/环境音频篡改）在 11 个月前已由 CompSpoof (arXiv 2509.15804) 提出，使得真正的新颖性仅限于音视频 + 场景一致性扩展；论文缺乏明确的数据/代码发布承诺以及人类评估锚点。
  • Full review: Claude Code 全文七公理审稿

Recent advances in speech synthesis and audio generation have made high-fidelity acoustic forgery low-cost and difficult to attribute, enabling a realistic attack scenario in which speech and background audio are independently manipulated over otherwise authentic video. Yet existing research either focuses on visual manipulation, addresses speech detection in isolation, or conflates speech and non-speech audio as a single undifferentiated audio stream, overlooking the distinct forensic challenges posed by background audio. This conflation is consequential: the two acoustic components arise from fundamentally different generative mechanisms, exhibit distinct artifact profiles, and pose different challenges to detection systems. We introduce MADBench, the first benchmark that treats speech and environmental audio as distinct acoustic components, enabling component-aware evaluation of audio deepfake detection across independently manipulated forgery sources. We benchmark representative state-of-the-art detectors and multimodal large language models under a unified protocol. Our experiments reveal that environmental audio manipulation is more detectable than synthetic speech across general-purpose encoders, while existing pretrained detectors fail on both acoustic components, and manipulated environmental audio asymmetrically degrades speech deepfake detection, findings entirely invisible under the single-label paradigm of prior benchmarks. MADBench establishes a rigorous foundation for future research into robust, component-aware audio deepfake detection.

2. AudioMap: Cloze-and-Choice Reinforcement Learning for Time-Aware Dense Audio Captioning

Authors: Yan Rong, Fengji Ma, Xu Li, Jinting Wang, Chen Zhang et al.

Categories: cs.SD Score: 6.60/10 (Obj:8 Id:6 Ind:5 Comp:6 Eff:8 Nov:8 Rep:6)

  • Strength: 7B 模型在 5 个公开 benchmark 上全面超越 30B 开源模型和 Gemini-3.1-Pro，TACOS temporal grounding 达 57.4（+15.9 vs Qwen3-Omni-Captioner-30B），cloze-as-reward 范式首次将 cloze test 从评测工具转为 RL 训练信号。
  • Weakness: 训练数据由 Gemini-3.1-Pro 生成且 reward judge (Qwen3.6-27B) 与评测 judge 部分重叠构成循环依赖风险；SFT 阶段 31.7 分提升中数据集贡献与方法贡献未分离，缺少同等数据上训练其他模型的对照实验。
  • Full review: Claude Code 全文七公理审稿

Time-aware dense audio captioning (TDAC) aims to generate multiple fine-grained attributes (dense) of the audio with precise time boundaries (time-aware). Existing methods struggle to achieve these two goals and mainly rely on supervised fine-tuning, yielding sub-optimal performance. While reinforcement learning (RL) shows promise, applying it to TDAC faces two main challenges: (1) existing rewards are too coarse to supervise multi-event, multi-attribute, and multi-relation descriptions in a fine-grained manner; and (2) temporal supervision is difficult for free-form captions, where flexible event-time expressions make reliable event-time correspondence challenging. To address these challenges, we propose AudioMap, a novel RL-based TDAC framework, which shifts to a unified cloze-and-choice reward paradigm. Specifically, we introduce the Evidence Sufficiency Reward (ESR) with an asymmetric hierarchical scoring mechanism to promote fine-grained accuracy and descriptive richness across diverse acoustic dimensions. Furthermore, we design the Event-Conditioned Temporal Reward (ECTR) to structurally bind timestamps to event semantics via temporal IoU, accompanied by a dual-curriculum learning strategy to facilitate the training process. Finally, to support this task, we construct the first time-aware fine-grained audio captioning dataset, AudioMapCap-44K, which contains 44K carefully annotated captions. Extensive experiments across diverse benchmarks show that AudioMap achieves state-of-the-art (SOTA) performance among open-source models and delivers competitive or superior results relative to proprietary models. Project page and release updates are available at https://github.com/ryysayhi/AudioMap.

3. REFRAMED: Towards Realistic Audio Description Generation for Movies

Authors: Igor Sterner, Mirella Lapata, Alex Lascarides, Frank Keller

Categories: cs.CL, cs.CV | COLM 2026 Score: 6.47/10 (Obj:9 Id:5 Ind:8 Comp:8 Eff:6 Nov:5 Rep:8)

  • Strength: 提出将 AD 生成从预设片段字幕重构为”描述什么+何时描述”的联合决策任务，并构建了 206 部电影/2,023 视频/双版专业 AD 的大规模数据集与多参考评测协议，暴露了 SOTA 模型与人类专家之间的巨大性能差距。
  • Weakness: 任务定义与 Mind the Gap 的 inline AD 概念有显著重叠，新颖性增量有限；实验未充分纳入 DistinctAD (CVPR’25) 和 NarrAD (WACV’25) 等最新 baseline 对比；what 与 when 两个子决策的消融隔离不足。
  • Full review: Claude Code 全文七公理审稿

Audio Description (AD) is a verbal narration of key visual content in videos, enabling access for visually impaired audiences. Unlike standard video captioning, AD is a structured editorial task: descriptions must be inserted into gaps in dialogue and must convey only what is needed to understand the narrative being told. However, existing approaches formulate AD generation in an artificial setting where both the content and timing of descriptions are pre-specified, reducing the task to clip-level captioning. They further rely on noisy transcription and alignment pipelines, and lack the rich parallel data required for modeling narrative context. We introduce a new formulation of AD generation in which models must jointly decide what to describe and when to do it. To support this, we present REFRAMED, a high-quality dataset of 2,023 videos that span 3,302 scenes from 206 movies, with professional AD transcripts (both American and British versions), professional subtitles and aligned screenplays. We also provide a manually curated challenge set that pairs full movies with multiple AD references, together with evaluation protocols that leverage dialogue gaps and multi-reference comparisons. Experiments with state-of-the-art AD systems and multimodal LLMs show that they outperform trivial baselines but fall far short of expert human performance. Our dataset and benchmark establish a new foundation for research on video understanding.

4. From Inaudible Inputs to Model Failures: Low-Frequency Safety Risks in LALMs

Authors: Yuanhe Zhang, Weiliu Wang, Jie Ren, Liang Lin, Zhenhong Zhou et al.

Categories: cs.SD, cs.AI Score: 6.42/10 (Obj:7 Id:6 Ind:8 Comp:5 Eff:6 Nov:7 Rep:6)

  • Strength: 首次提出低频（5–20 Hz）不可听信号作为 LALM 攻击向量，在 6 个 LALM 上实现全部 18 个 model-task cell 精度下降（mean 13.1pp, max 67pp），ANR 仅 0.06–0.08%，人类听觉评分 1.33 接近 clean 的 1.17。
  • Weakness: 实验为纯数字模拟无物理空间验证；缺少最密切相关工作 Sirens’ Whisper (arXiv 2603.13847, USENIX Security 2026) 的引用与对比；ILL vs Gaussian/PNL 的攻击效果差异在 Holm 校正后不显著（p=0.422），核心贡献更多在于隐蔽性而非攻击力增量。
  • Full review: Claude Code 全文七公理审稿

Large audio-language models (LALMs) have demonstrated strong capabilities in understanding diverse audio inputs. This diversity includes low-frequency signals that are inaudible to humans but can still enter the model and influence its generation. However, the practical impact of such low-frequency inputs on LALMs remains largely unexplored. In this paper, we propose Intermittent Low-Frequency Lockout (ILL), an inaudible red teaming method that evaluates this risk using a universal waveform template in a black box setting. ILL uses Sentence Attention Scale Estimation to determine active intervals and Frequency Confusion Transfer to construct a low-frequency waveform with continuous phase from corpus spectral variation. To mitigate this risk, we propose Distributional Requery Guard (DRG) to detect low-frequency distribution shifts and conditionally request a second recording for semantic recovery. Across six LALMs and multiple audio understanding tasks, ILL reduces accuracy by up to 67 percentage points while receiving a mean human audibility rating of 1.33, close to 1.17 for clean audio; DRG raises mean attacked accuracy from 28.5\% to 46.1\% after clean reacquisition. These findings identify a previously overlooked safety risk for LALMs and provide a foundation for future research on robust audio understanding.

5. Dynamic Clustering for Cross-Segment Permutation Alignment in Long Speech Separation

Authors: Yuzhu Wang, Archontis Politis, Konstantinos Drossos, Tuomas Virtanen

Categories: eess.AS, cs.SD | 6 pages, accepted by MLSP 2026 Score: 6.42/10 (Obj:9 Id:6 Ind:9 Comp:5 Eff:8 Nov:5 Rep:6)

  • Strength: Training-free 动态聚类方法在稀疏长语音场景取得 96.8% 排列准确率和 13.3 dB SI-SDR，比最强 baseline 提升 7.5% / 5.1 dB，接近 oracle 上界（14.3 dB），且对说话人数估计误差具有鲁棒性。
  • Weakness: 核心 idea（动态说话人嵌入参考池 + 相似度匹配排列对齐）与 streaming diarization 中的 STB/AOSC 方法高度相似，增量贡献为工程适配而非机制创新，且缺乏 TopN 池限制本身的消融对照和代码开源承诺。
  • Full review: Claude Code 全文七公理审稿

Long speech separation typically employs a segment-separation-stitch paradigm where recordings are divided into short segments, processed independently, and stitched together. Its challenge lies in predicting cross-segment permutations. This paper proposes a training-free dynamic clustering approach for cross-segment permutation alignment using speaker embedding reference pools. The method predicts the permutation using the cosine similarity between current segment embeddings and the reference pools. The approach updates reference pools by retaining the most representative speaker embeddings based on their overall cosine similarity with existing references. As a plug-and-play post-processing module compatible with existing separation models, the proposed method demonstrates superior performance compared to existing methods on dense and sparse long speech scenarios, particularly in challenging sparse scenarios with extended utterance gaps, and further shows robustness to speaker count estimation errors in unknown speaker count scenarios.

6. Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions

Authors: Oluwanifemi Bamgbose, Simon Rosen, Jash Shah, Lindsay Devon Brin, Hoang H Nguyen et al.

Categories: cs.SD, cs.AI, cs.CL | Work in progress Score: 6.40/10 (Obj:8 Id:6 Ind:5 Comp:8 Eff:6 Nov:7 Rep:8)

  • Strength: 首个语言学扎根的 10 维 TTS 元评估基准，用训练有素的语言学家标注（Krippendorff α 0.46-0.82），系统揭示了 MOS 预测器坍缩到信号质量、Audio-LLM 评判模型存在 prompt 依赖性检测盲点（如 C2b 条件下 2/4 模型输出坍缩）。
  • Weakness: 860 样本规模偏小（最少维度仅 46 样本），单一 TTS 架构（Cartesia Sonic-3），32.3% 设计错误未被人类标注者感知，且 2/4 Audio-LLM 评判模型为闭源 API，限制了发现的可推广性和完全可复现性。
  • Full review: Claude Code 全文七公理审稿

Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Models (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive. We deconstruct “naturalness” into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters. Results from benchmarking four MOS predictors and four Audio-LLM judges reveal that MOS predictors collapse onto acoustic signal quality, while Audio-LLM judges show selective, prompt-dependent detection that does not generalise across all dimensions. Neither class reliably captures a breadth of linguistically structured speech errors. Our dataset, annotation schema, and evaluation code are publicly released to support more targeted and interpretable TTS evaluation.

7. Comparing British and American Audio Description of Movies

Authors: Igor Sterner, Alex Lascarides, Frank Keller

Categories: cs.CL | CMN 2026 Workshop Score: 6.40/10 (Obj:8 Id:8 Ind:6 Comp:8 Eff:6 Nov:5 Rep:5)

  • Strength: 首次在206部电影的配对语料上定量验证了英美音频描述在7个语言学维度上的显著差异（全部p<0.01），填补了此前仅有定性分析的空白。
  • Weakness: 方法全为现成工具的标准应用、7个假设全部来自现有指南且全部为真（无反直觉发现）、效应量普遍很小（如被动语态4.6% vs 1.3%），且核心语料库的可获取性和代码开源情况未明确。
  • Full review: Claude Code 全文七公理审稿

Narrating the visual component of movies is known as audio description. It is a narrative technique designed to enable blind and visually impaired individuals to follow the story. However, it is far more constrained than most narratives: the descriptions not only need to convey the story in the movie, but they must also fit into gaps between dialogue and they need to conform to guidelines that exist in each region. In this work, we compare audio description created in the United Kingdom against audio description created in the United States. We use guidelines written for these two regions, alongside the impressions from a practitioner in the field, to motivate specific hypotheses about the differences. We test these hypotheses against our pre-existing corpus, which provides both human-authored American and British audio description for each of 206 movies. Results provide quantitative evidence to uphold all tested hypotheses, including differences in lexicon, the use of the progressive aspect, the use of passive constructions, the use of subjective adjectives and modifiers, when characters are named, how scenes are cued, and degree of overlap with movie dialogue and music. Our work offers a quantitative lens into the narrative technique of audio description.

8. CodecArena: Codec Quality Assessment via Visual Reinforcement Learning

Authors: Jiaye Fu, Weiqi Li, Qiankun Gao, Yanchen Zhao, Xiandong Meng et al.

Categories: cs.CV | The project page is: https://jyfu-vcl.github.io/codecarena Score: 6.20/10 (Obj:8 Id:6 Ind:5 Comp:6 Eff:8 Nov:6 Rep:6)

  • Strength: 在 80 视频 source-disjoint 人类标注 benchmark 上达到 SRCC 0.895 / pairwise accuracy 94.9%，大幅超越 Q-Insight/VQ-Insight 等 VLM 评估器和 LPIPS/DISTS，首次将编解码器评测重构为 source-conditioned facet-level 推理。
  • Weakness: 训练标签由 VLM Ensemble (Kimi-K2.6 + Opus-4.7) 生成且评测 baseline 使用同一组 VLM，存在模型家族重叠的循环风险；80 视频 benchmark 规模偏小且代码/数据开源承诺未明确。
  • Full review: Claude Code 全文七公理审稿

Video coding is advancing into the low and ultra-low bitrate regime, driven by end-to-end codecs that replace the hand-crafted pipeline with jointly optimized neural networks and generative codecs that exploit the priors of video generation models. Yet the dominant metrics, LPIPS and DISTS, measure feature and texture similarity rather than content fidelity: a reconstruction that hallucinates a wrong face or blurs text into convincing strokes can still score well, even when a human rejects it instantly. To address this, we propose CodecArena, the first vision-language framework for video coding quality assessment, casting codec evaluation as source-conditioned comparative reasoning between a reference and its reconstructions. We optimize CodecArena with Facet-GRPO, a visual reinforcement learning scheme that aligns pairwise codec preferences while grounding the verdict in five fidelity facets: identity, objects, text, texture, and temporal consistency. Its facet-anchored reward uses automatically derived facet directions as weak anchors, rather than human per-facet labels, to prevent any single sub-score from dominating the holistic preference and to yield interpretable fine-grained quality judgments. To support training and evaluation in this underexplored regime, we construct two complementary resources: CodecArena-1K, a fully automatic preference dataset of 1,500 comparison groups built from traditional, neural, and generative codec reconstructions with fused vision-language and objective supervision; and CodecArena-Bench, a human-ranked benchmark with source-disjoint videos for fair out-of-domain evaluation. Extensive experiments demonstrate that CodecArena achieves state-of-the-art agreement with human judgments on source-disjoint content across diverse codecs and bitrates, surpassing perceptual metrics and prior vision-language evaluators.

9. Beyond Piano: Cross-Instrument MIDI Velocity Estimation via Differentiable SoundFont Proxies

Authors: Zhanhong He, Hanyu Meng, David Defeng Huang, Roberto Togneri

Categories: eess.AS | Accepted to ISMIR2026 conference Score: 6.00/10 (Obj:7 Id:6 Ind:5 Comp:8 Eff:4 Nov:6 Rep:8)

  • Strength: 提出 Diff-SFProxy 通过 loudness-related 参数（PHE/OSF）而非波形重建监督 velocity，在 guitar 上 rBSTL 较 zero-shot transfer 提升 +0.062~+0.089，且对 backend crop length 变化稳定（Δ=0.006 vs Diff-Synth 的 0.083）。
  • Weakness: 仅验证 piano 和 guitar 两种乐器，guitar 上提升幅度有限（rBSSL +0.025~+0.037 over pretrained），缺少外部 baseline 比较，关键 Diff-Synth vs Diff-SFProxy 比较同时混淆了后端架构/训练数据/损失域三个变量。
  • Full review: Claude Code 全文七公理审稿

Many music datasets contain MIDI notes but lack reliable velocities, defaulting to a constant value. This absence is especially problematic outside the piano domain, as velocity is a core component for expressive rendering, music generation, and performance analysis. This paper studies cross-instrument MIDI velocity estimation in this label-scarce setting. Starting from a piano-trained velocity estimator, we recast target-instrument adaptation as predicting renderer-conditioned velocities whose rendering matches the dynamics of the performance audio. This adaptation can be driven by either differentiable synthesizers (Diff-Synth) or our proposed differentiable SoundFont proxies (Diff-SFProxy). We highlight the Diff-SFProxy: it supervises velocity through note-wise, loudness-related acoustic parameters rather than waveform reconstruction, focusing gradients on velocity-dependent behavior. Experiments on piano and guitar show that Diff-SFProxy is effective for cross-instrument MIDI velocity estimation, while waveform-domain Diff-Synth degrades performance.

10. MusicLayout: Explicit Structural Planning for Controllable Text-to-Music Generation

Authors: Shuyu Li, Kejun Zhang, Jiahe Lei, Shulei Ji, Zihao Wang et al.

Categories: cs.SD, cs.AI, cs.MM Score: 5.79/10 (Obj:8 Id:8 Ind:6 Comp:6 Eff:5 Nov:5 Rep:6)

  • Strength: Matched-data 三层渐进消融（no-layout / shuffled-training / shuffled-inference）有效隔离了显式布局规划的因果贡献，在 FreeMIDI 和 MidiCaps 上 reference-layout 相对 no-layout control 全部 7 个指标均改善。
  • Weakness: 在唯一真实音频评测集 MuChin 上 FAD (3.456) 远差于所有 baseline（最佳 1.994），且未与 SegTune (ACL’26 Oral) 和 MusicWeaver 两篇已发表的高度相关工作进行比较，核心 plan-then-generate 范式直接迁移自 PlanGen 而无新机制创新。
  • Full review: Claude Code 全文七公理审稿

Text-to-music generation has advanced rapidly, but current systems still rely primarily on global text prompts, leaving the structural organization of generated music implicit and difficult to inspect, control, or revise before audio generation. To address this issue, we introduce MusicLayout, an explicit intermediate representation for controlling musical structure in text-to-music generation. MusicLayout describes a musical piece as a time-aligned layout of sections, textures, repetitions, variations, and instrument-level arrangements, serving as an interpretable planning layer between textual intent and the generated music. We integrate MusicLayout into a text-to-music framework built on a unified autoregressive formulation, where the model first generates a MusicLayout representation and subsequently predicts audio tokens conditioned on this representation within a single sequence. The resulting MusicLayout can be inspected and modified prior to audio generation, providing a mechanism for layout-level structural control. We evaluate MusicLayout through layout-conditioned generation, layout manipulation experiments, and matched-data ablations, providing evidence that explicit layout planning can improve long-range structural organization and support layout-level control.

11. SonicWeave: Chunk-Routed Mixture-of-Experts for Unified Audio Scene Generation

Authors: Yunrui Cai, Xu Li, Yucheng Zhou, Jinchao Li, Dingdong Wang et al.

Categories: cs.SD Score: 5.70/10 (Obj:8 Id:6 Ind:5 Comp:6 Eff:6 Nov:5 Rep:4)

  • Strength: CPE-MoE 的 chunk-level prior-evidence routing 在统一音频生成中实现了 TTS 并列最佳 (WER 1.0%/CER 0.8%) 和 Complex-Scene MOS-R 4.49（比最强外部 baseline +2.07），routing 分析清晰展示了内容依赖和 diffusion-phase 自适应的专家分工。
  • Weakness: 公开 benchmark (AudioCaps/MusicCaps) 上未取得任何单项 SOTA，最强结果依赖自建 100-prompt Complex-Scene suite + 闭源 Gemini judge + 25 人小样本 human study，且训练数据为不可获取的内部 20k+ 小时语料，核心实验不可独立复现。
  • Full review: Claude Code 全文七公理审稿

Text-conditioned general audio generation is moving beyond isolated speech, music, and sound-effect synthesis toward a single model that can compose them into controllable, coherent audio scenes. This unified setting is particularly challenging: heterogeneous components impose conflicting structural requirements on a shared backbone, while a complex mixed scene may contain locally distinct or overlapping content that demands fine-grained adaptation within the same clip. Existing audio mixture-of-experts (MoEs) mainly route at the domain level, while token-wise routing overlooks the local continuity inherent to acoustic signals. We propose SonicWeave, a flow-matching model for unified audio scene generation. At its core is a chunk-routed MoE with a conflict-gated prior-evidence routing mechanism (CPE-MoE). CPE-MoE routes contiguous acoustic chunks by combining a global prior that encodes the structured text condition and diffusion phase with local evidence from the evolving acoustic state. A learned conflict gate favors the prior when local states are unreliable, while allowing local evidence to influence routing when a region departs from the global scene context. SonicWeave supports speech, music, sound effects, singing, and their fine-grained mixtures with a single set of weights. Across TTS, TTA, and TTM benchmarks, SonicWeave consistently improves over controlled Dense and Base-MoE baselines. Complex-scene evaluation further demonstrates improved compositional quality, while routing analyses reveal content-dependent expert specialization across diffusion phases. These results suggest that temporally coherent, prior-evidence routing is an effective conditional-computation strategy for unified audio generation. Project page: https://caiyunrui.github.io/SonicWeave.

12. RAG-Audio: Retrieval-Augmented Generation for Faithful Brain-to-Audio Reconstruction

Authors: Ambuj Mehrish, Sebastiano Vascon

Categories: cs.SD, cs.AI Score: 5.68/10 (Obj:6 Id:8 Ind:5 Comp:5 Eff:6 Nov:4 Rep:6)

  • Strength: Autoregressive negative control (MusicGen 0.18→0.20 vs AudioLDM 0.14→0.43) 有效隔离了 latent-trajectory initialization 作为机制，FAD 降低约 10.8× (13.49→1.25)。
  • Weakness: 方法本质是 SDEdit + kNN 检索的直接组合，仅在单一小数据集（5 名被试，10 类音乐）上验证，代码仓库标注 TBA 不可用，且未与最近的同类工作 Ciferri et al. (2025) 进行实验比较。
  • Full review: Claude Code 全文七公理审稿

Brain-to-audio reconstruction is limited by \emph{prior domination}: when a pretrained generator is conditioned on a weak neural signal, it produces realistic but stimulus-inaccurate audio. We introduce RAG-Audio, which decodes fMRI into a semantic audio embedding, retrieves a matching real-audio exemplar, and initializes the frozen generator’s sampling trajectory from that exemplar while retaining the decoded embedding as conditioning. On Brain2Music, RAG-Audio improves 10-way stimulus identification from $0.14$–$0.18$ for direct generation, near the $0.10$ chance level, to $0.40$–$0.43$, comparable to retrieval. It also reduces Fréchet Audio Distance by roughly an order of magnitude, from $13.49$ to $1.25$ for AudioLDM. RAG-Audio approaches nearest-neighbor retrieval in identification while remaining generative; its higher FAD is expected because retrieval directly replays real audio. An autoregressive negative control, which lacks an initializable latent trajectory, shows no comparable gain, attributing the improvement to trajectory initialization. These results suggest that retrieval-guided initialization can mitigate prior domination in brain-to-audio generation.

13. EmoS: A Theory-Grounded Framework for Evaluating and Aligning Emotional Intelligence in Spoken Language Models

Authors: Junyu Wang, Siyuan Zhang, Peiyuan Jiang, Jian Zong, Jingyu Zhang et al.

Categories: cs.CL | Accepted at ACM Multimedia 2026 (MM ‘26) Score: 5.58/10 (Obj:8 Id:6 Ind:4 Comp:5 Eff:6 Nov:5 Rep:5)

  • Strength: 基于 Mayer-Salovey 四分支模型构建首个覆盖感知-理解-使用-管理全链路的 SLM 情感智能评测框架，EmoS 评估器在合成 benchmark 上达 83.8%（接近人类 86.0%），在真实 YouTube 数据上达 62.9%（优于最强 baseline Gemini 2.5 Pro 19.6 pp）。
  • Weakness: 训练数据、测试集标签和 RFR judge 全部来自 LLM 生成链（DeepSeek-R1 + Qwen3-8B），主测试集 EmoSBench 的 ground truth 缺乏独立人类标注，构成 major 级循环论证风险；且 EchoMind（2025-10）和 EMO-Reasoning（2025-08）等非同期前置工作削弱了 “first” novelty claim。
  • Full review: Claude Code 全文七公理审稿

Despite significant advances in instruction-following and auditory comprehension, the evaluation of Emotional Intelligence (EI) in Spoken Language Models (SLMs) remains confined to rudimentary paralinguistic perception, lacking a systematic, theory-driven cognitive framework. We introduce EmoSBench, the first comprehensive EI evaluation benchmark for SLMs constructed upon the four-branch theoretical model, covering Perceiving, Understanding, Using, and Managing Emotion across ten sub-tasks. Preliminary assessments on EmoSBench reveal a substantial gap: even leading proprietary models like GPT-4o-Audio achieve only 52.6%, significantly trailing human baselines. To bridge this gap, we develop EmoS, a specialized evaluator model optimized via Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO). To facilitate its effective training, we curate EmoDialogue, a bilingual dataset providing necessary fine-grained supervision through response pairs with rigorously defined EI gradations. Concurrently, we introduce a reward mechanism integrating a Steep Exponential Accuracy Reward (SEAR) and a Rationale Fidelity Reward (RFR) to enforce precise ordinal scoring and valid reasoning. Experiments demonstrate that EmoS reaches 83.8% accuracy, approaching human-level performance. Furthermore, evaluations on authentic, unconstrained spoken interactions validate its robust real-world generalization, establishing a foundational framework for advancing emotionally intelligent dialogue systems.

14. Structured Phonological Representations for Audio-Articulatory rtMRI Speech Classification

Authors: Abner Hernandez, Tomás Arias Vergara, Daiqi Liu, Andreas Maier, Paula Andrea Pérez-Toro

Categories: cs.CL, cs.SD, eess.AS | Submitted for review at SLT 2026 Score: 5.50/10 (Obj:8 Id:5 Ind:6 Comp:5 Eff:5 Nov:4 Rep:7)

  • Strength: PhonoQ 表示与 WavLM 融合在 unseen-speech 下将 39-phoneme macro-F1 从 64.0% 提升到 69.0%，并在五个音系目标上全面领先，后验分析展示了可解释的 flapping/affrication/nasal assimilation 模式。
  • Weakness: Contour-only 教师监督增益微弱（+2.1~+2.3 macro-F1），未与同组最近竞品 Liu 2025 (arXiv 2507.17682) 和 Park 2026 (ICASSP) 定量对比，PhonoQ vs SSL 比较未控制参数量/训练数据等混淆变量，新颖性限于 PhonoQ 表示向 rtMRI 场景的直接迁移。
  • Full review: Claude Code 全文七公理审稿

Real-time MRI makes it possible to observe vocal-tract articulation during speech, but mapping these articulatory patterns to phonetic and phonological categories remains challenging. We investigate whether PhonoQ, an audio-based model trained to recognize structured phonological features, provides useful information for audio–articulatory modeling. Specifically, we extract representations from PhonoQ’s Conformer module, whose training is shaped by supervision for manner, place, voicing, and vowel features. Using articulatory contours with synchronized audio-derived features, we compare WavLM-large and HuBERT-large baselines with models that incorporate PhonoQ-derived representations. Across unseen-speech and unseen-subject settings, these features improve macro-F1 for phonological targets including manner, place, voicing, vowel height, and vowel backness, and also improve fine-grained 39-phoneme classification. In a contour-only inference setting, audio-derived teacher supervision yields modest but consistent gains over contour-only training, indicating that phonological information from synchronized audio can be partially transferred to articulatory models. Finally, posterior analyses show interpretable surface-sensitive patterns consistent with flapping-like /t/ realizations, /t/-/r/ retraction or affrication, and nasal place assimilation.

15. Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models

Authors: Zhi Zeng, Cheng Zhang, Zesheng Yang, Rendong Pi, Jiaying Wu et al.

Categories: cs.AI Score: 5.47/10 (Obj:8 Id:5 Ind:4 Comp:5 Eff:6 Nov:5 Rep:5)

  • Strength: 提出 ST-OmniQA benchmark（40K 全景视频 + 400K QA 对，四级能力 + modality-necessity 约束 + executable reasoning graphs），填补了时空音视觉推理评测的空白；ST-Omni-R1 在该 benchmark 上达 77.83% 平均准确率，较最佳 baseline 提升 40.55pp，并在三个真实数据集上表现出迁移能力。
  • Weakness: 关键 baseline 遗漏（SPUR、Spatial-Omni、OWL 未评估）；通用模型零样本 vs 微调模型比较不公平；LLM judge (DeepSeek-v4-flash) 缺乏独立人类校验；STA-encoder 核心组件（trajectory tokens vs semantic tokens）未消融；代码/数据/checkpoint 未承诺开源；音频分支核心 idea 与并发工作 ST-AudioLM 高度重叠。
  • Full review: Claude Code 全文七公理审稿

Understanding dynamic sound sources requires jointly determining what produces a sound, where the source is located, and how it moves over time. Yet existing audio-language models often represent clips as global acoustic events, while vision-language models lack the spatial audio cues needed to localize and track individual sources. To evaluate this missing capability, we introduce ST-OmniQA, a spatio-temporal audio-visual question-answering benchmark built from panoramic videos paired with synchronized first-order Ambisonics (FOA) audio of moving sound sources. It contains 40K videos and 400K question-answer pairs organized into four capability levels covering sound-event recognition, direction of arrival, source distance, motion trajectories, and temporally grounded audio-visual reasoning. Building on this benchmark, we propose ST-Omni-R1, which integrates FOA-derived semantic and trajectory representations with panoramic visual context and is trained through progressive curriculum learning and reasoning-tree reinforcement learning. ST-Omni-R1 achieves 77.83\% average semantic accuracy across the four levels, compared with 37.28\% for the best evaluated baseline. Results on three public spatial-audio benchmarks further indicate that its learned spatial and motion representations transfer beyond ST-OmniQA.

16. Agreement-Based Audio-Visual Segmentation:Champion Report for the MeViS-Audio Track in the 8th LSVOS Challenge

Authors: Yiwen Ren, Jianing Liu, Yingxin Wang, Kexin Zhang, Licheng Jiao et al.

Categories: cs.CV Score: 5.16/10 (Obj:5 Id:4 Ind:8 Comp:5 Eff:6 Nov:4 Rep:5)

  • Strength: 赢得 ECCV 2026 8th LSVOS MeViS-Audio track（Final 0.769589），agreement-based selection 提供了一种 label-free 的跨模型掩码轨迹选择方法，在 100-query 消融中 J&F 从 74.60 提升至 75.15。
  • Weakness: 系统使用 8+ 开源模型拼装但缺乏与前届系统的性能对比和充分的组件级消融，hidden test J&F 仅 0.5952 且较 validu 下降 17 个百分点，novelty 主要限于已有的 ensemble medoid 选择在新领域的应用。
  • Full review: Claude Code 全文七公理审稿

The MeViS-Audio track asks a system to segment the objects described by a spoken motion expression throughout a video and to return empty masks when the described target is absent. We present a simple staged solution. Qwen3-ASR first converts speech into text. Several video mask tracks are then produced with complementary grounding and segmentation models. Instead of trusting a single prediction, we select the track that has the highest average mask agreement with the other candidates. A small set of explicit direction, count, and plural rules corrects queries that require more than ordinary single-object tracking. Finally, a video-level classifier combines visual, audio-visual, and within-video query scores to decide whether any target is present. The submitted system obtains 0.5952 J &F, 0.7931 no-target accuracy, 0.9205 target accuracy, and a final score of 0.769589. The challenge organizers notified our team that this result ranked first in the track.

17. AI-Guided Learning: Research on Knowledge and Skill Acquisition Support Methods Using Deep Learning Audio-Video Processing Techniques

Authors: Kazuki Kawamura

Categories: cs.HC, cs.MM | 118 pages, 26 figures, 4 tables. Doctoral dissertation, Doctor of Interdisciplinary Informatics, The University of Tokyo; degree awarded September 19, 2025 Score: 5.10/10 (Obj:7 Id:5 Ind:7 Comp:4 Eff:6 Nov:3 Rep:5)

  • Strength: AIxSpeed 在 LibriSpeech/UME-ERJ 上实现 1.30x/1.29x 内容自适应加速且 MOS 优于恒定倍速，FastPerson 减少 53% 观看时间且 quiz 分数无显著差异，两项结果的绝对效用清晰可量化。
  • Weakness: 本文为三篇已发表工作（UIST 2022 / AugHuman 2024 / arXiv 2022–2026）的学位论文 compilation，无技术增量；Profy 的发音改善证据极弱（无对照组、无效应量、无样本量），且三者未与领域 SOTA（HiPPO、BiCAPT、Video-MMLU 等）对比。
  • Full review: Claude Code 全文七公理审稿

Audio and video have become major learning media, but learners face two persistent challenges: the time cost of consuming long-form content sequentially and the lack of scalable feedback for imitation-based skill acquisition. This dissertation proposes an AI-guided learning framework that supports three interconnected stages: Consume, Understand, and Imitate. It develops and evaluates three systems. AIxSpeed dynamically adjusts audio playback speed at the phoneme level using speech-recognition-model confidence as a proxy for listening difficulty. FastPerson generates multimodal video summaries that preserve visual and auditory information and lets learners switch between summarized and full versions by chapter. Profy learns proficiency from largely unannotated speech data and visualizes classifier-relevant regions and model-derived acoustic distances to support pronunciation practice. Technical and user evaluations show that AIxSpeed achieved average playback factors of 1.30x on LibriSpeech and 1.29x on UME-ERJ and received higher mean opinion scores than matched constant-speed playback; FastPerson reduced viewing time by 53% with no statistically significant difference in quiz scores compared with normal playback; and Profy showed an observed improvement in pronunciation intelligibility, with non-overlapping pre- and post-practice confidence intervals. Together, these systems demonstrate how deep learning can support efficient content consumption, multimodal understanding, and repeated skill practice while retaining learner access to the original material.

18. Neural Array-Generic Direction-of-Arrival Estimation Exploiting Array Transfer Functions

Authors: Mikko Heikkinen, Archontis Politis, Konstantinos Drossos, Tuomas Virtanen

Categories: eess.AS | Accepted for publication in IWAENC 2026 Score: 4.95/10 (Obj:8 Id:5 Ind:6 Comp:5 Eff:4 Nov:5 Rep:5)

  • Strength: 提出 ATF 替代几何坐标作为阵列描述符，在 3D 多源场景中比 MUSIC 退化更缓慢（4 源时 LE 34.1° vs 53.4°），并展示了对未见手机类阵列的泛化能力。
  • Weakness: 缺少 ATF vs. 几何坐标的消融对照和关键 baseline（Neural-SRP/GI-DOAEnet/PhaseCoder）对比，2D 任务输给全部 baseline，3D 单源场景 LE 差距大（22.9° vs MUSIC 8.4°），且未在真实测量数据上验证。
  • Full review: Claude Code 全文七公理审稿

Direction-of-arrival (DoA) estimation is a key component of multichannel audio processing, yet many deep learning approaches remain tied to the microphone arrays used during training and generalize poorly to unseen devices. This paper proposes an array-generic neural DoA estimation framework using measured or simulated complex directional array transfer functions (ATFs) matched to real-world multi-microphone devices. The method processes multichannel spectrograms and ATF metadata with separate convolutional encoders, fuses the resulting representations through cross-attention, and predicts source directions using a multi-source Cartesian vector output formulation. Experiments on simulated 2D and 3D localization tasks under reverberation and diffuse babble noise show that the proposed approach generalizes to previously unseen arrays, including mobile-phone-like configurations, without major performance degradation, while remaining competitive with conventional and learning-based baselines.

19. A Multi-Scale Temporal Framework with Dynamic Fusion for EEG-Based Emotion Recognition

Authors: Stefanos Gkikas, Yang Guo, Guangliang Li, Raul Fernandez Rojas, Giorgos Giannakakis et al.

Categories: cs.AI, cs.LG, cs.SD | The paper has been accepted at: IEEE | 2026 9th International Conference on Pattern Recognition and Artificial Intelligence (PRAI 2026) Score: 4.95/10 (Obj:6 Id:5 Ind:8 Comp:5 Eff:4 Nov:4 Rep:6)

  • Strength: 在 DMER 数据集上进行了系统性的 29 种配置搜索，受试者独立评估设计严谨（分层难度平衡分割），三尺度动态融合在二分类任务上达到 65.22%（全信号基线 57.41%，+7.81pp）。
  • Weakness: 无任何外部基线对比（未引用 DMER 原论文基线、DGCNN、EEG-Conformer、miMamba、MSGM 等同类方法），三分类最佳 45.43% 仅比随机水平高 12pp，动态融合相对拼接在三分类仅优 0.12pp（13 个测试受试者，统计意义存疑），多尺度+注意力+动态融合各组件均为已有方法。
  • Full review: Claude Code 全文七公理审稿

Mixed emotions represent a clinically relevant but still underexplored target for automatic emotion recognition. EEG provides millisecond-level access to neural activity, yet most EEG pipelines analyze the signal through a single temporal window, thereby fixing the temporal structure available to the model. This study introduces a multi-scale temporal framework for EEG-based emotion recognition. The EEG waveform is decomposed into windows of one or several durations, processed by a shared attention-based encoder, and integrated through a dynamic fusion module that assigns sample-specific weights across temporal scales. The framework is evaluated under a subject-independent protocol in binary and three-class settings, with the three-class task including the mixed affective category. The best results are 65.22% for the two-class task and 45.43% for the three-class task. Both are obtained with three-scale dynamic-fusion configurations and remain substantially above the full-signal baseline. The best-performing temporal scales differ between the two tasks. Dynamic fusion outperforms concatenation in the highest-scoring two-class configuration and slightly exceeds it in the highest-scoring three-class configuration, although these multi-scale settings require substantially more computation than the full-signal baseline.

20. Speaker Role and Language Diarization for Analyzing Multilingual Interviews for Language Proficiency of Older Adults

Authors: Anfeng Xu, Tiantian Feng, Kevin Huang, Pranali Khobragade, Sudarsana Kadiri et al.

Categories: eess.AS | Under Review Score: 4.74/10 (Obj:6 Id:5 Ind:6 Comp:5 Eff:5 Nov:4 Rep:5)

  • Strength: 在真实 multilingual older-adult interview 场景中验证了 Whisper-based speaker-role + language diarization pipeline，并发现 simple diarization-derived behavioral features（speech ratio, intended language ratio）与 Whisper embeddings 在 pro。
  • Weakness: 数据规模极小（9.71h 音频，仅 Hindi 158 条做 prediction），language diarization 最佳 LC 仍达 24.93%，proficiency prediction 绝对性能有限（PCC=0.531, binary acc=65.82%），且未与 DISPLACE challenge 系统、wav2vec 2.0 proficiency baseline 或 Speech LLM baseline。
  • Full review: Claude Code 全文七公理审稿

Automatic language proficiency assessment in the context of multilingual interview-based settings remains underexplored. In this work, we develop Whisper-based speaker-role and language diarization systems to automatically extract respondent speech and characterize language usage in multilingual interviews with older adults. We further investigate whether diarization-derived conversational and language-use behaviors can support downstream language proficiency assessment. Results show that language-adapted Whisper models substantially improve language diarization performance for lower-resource and linguistically related Indian languages. Statistical analyses reveal that respondent speech ratio and intended language usage are strong predictors of proficiency ratings. Furthermore, simple diarization-derived behavioral features achieve performance comparable to Whisper-based speech embeddings for proficiency prediction, while combining both yields the best results. Importantly, both the speech and language use statistical analyses and language proficiency prediction performance remain largely preserved when using fully automatic diarization outputs, demonstrating the potential of respondent-centric conversational analysis for scalable language proficiency assessment.

21. Towards Expressive and Faithful Audio-to-Image Generation: A Unified Multimodal Dataset and Synthesis Framework

Authors: Dongxu Ge, Shansong Liu, Cheng Gong, Xiao-Lei Zhang, Chi Zhang et al.

Categories: cs.CV | 23 pages, 16 figures Score: 4.74/10 (Obj:6 Id:5 Ind:3 Comp:5 Eff:5 Nov:4 Rep:8)

  • Strength: 构建 323K 三模态数据集 A2I-Set，用 10+ 模型串联 pipeline 做质量过滤，AudioCanvas 在美学偏好指标 (Aes=5.47, HPSv3=4.8115) 和对齐指标 (AIS=0.0841, TIS=0.2547) 上领先 baseline。
  • Weakness: 评测集与训练集同源（论文自承），60% 训练图像由 FLUX 生成导致美学指标评测偏向，核心生成质量指标 FID 在两个测试集上均未取胜（A2I-eval: 73.93 vs AudioToken 69.79; Landscape: 137.03 vs 100.42），方法为已有组件直接拼接无新机制。
  • Full review: Claude Code 全文七公理审稿

As an important subfield of cross-modal generation, synthesizing static visual content in the form of images from audio, namely audio-to-image (A2I) generation, has attracted increasing research attention in recent years. Nevertheless, despite the remarkable visual quality of modern text-to-image (T2I) models, the performance of A2I remains fundamentally limited by traditional datasets, which often lack both high-fidelity images and precise cross-modal alignment. As a result, existing methods still struggle to achieve high-quality audio-to-image generation through finetuning strong T2I models, thereby constraining practical applications in this area. Motivated by this gap, we introduce A2I-Set, a unified, high-quality tri-modal dataset consisting of 323K paired audio, images, and detailed text captions, specifically designed for audio-visual research, including audio-conditioned image generation. Besides, we developed a new mixed-source test set for the A2I task through human supervision. We further propose an A2I model, AudioCanvas, fine-tuned on our A2I-Set. Experiments show that AudioCanvas achieves more visually expressive as well as cross-modal alignment results that generally outperforming existing approaches. Our dataset and source code are available at https://github.com/gdx012/A2I-Generation.

22. DAVE: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation

Authors: Wei Zhou, Wanyi Ning, Yinshang Guo, Qianxiao Fang, Haitao Qian et al.

Categories: cs.SD, cs.AI Score: 3.84/10 (Obj:6 Id:5 Ind:3 Comp:3 Eff:5 Nov:2 Rep:5)

  • Strength: 完整的 challenge 竞赛系统，整合数据增强（219K mixtures）、多目标训练、解耦视觉归因和选择性后处理，在 Real-World AVSE Challenge 两个赛道均大幅超越官方 baseline 并进入前 5 名。
  • Weakness: 七个评测指标中至少五个的训练 loss 与评测模型直接共享（同一 ASR/speaker/UTMOS 模型），构成 Goodhart’s Law 循环论证；核心”decoupled”设计缺少 coupled baseline 对比；所有组件均为已有工具拼装，无技术创新；代码和数据未开源。
  • Full review: Claude Code 全文七公理审稿

Audio-visual speech enhancement under real-world conditions remains challenging due to unreliable visual inputs and the lack of large-scale training data with realistic acoustic conditions. Existing approaches usually fuse visual features directly into the separation network, making them vulnerable to degraded visual signals. In this paper, we present DAVE, a decoupled audio-visual enhancement framework for real-world speech separation. Firstly, to address the data scarcity issue, we construct DAVE-Corpus, a large-scale training corpus with 219,411 mixtures generated from public meeting corpora through combinatorial acoustic augmentation. Then, we introduce a progressive multi-objective optimization strategy to jointly improve speech separation, intelligibility, speaker identity preservation, and perceptual quality. We further develop a certified selective enhancement chain that applies scene routing, GAN-based denoising, and loudness normalization only within the no-reference partition, guaranteeing non-degradation of reference-based metrics. Experimental results on the Real-World Audio-Visual Speech Enhancement Challenge demonstrate the robustness of DAVE under both real-world mixed scenarios and visual degradation conditions.