OpenMOSS Releases MOSS-Audio: An Open-Source Foundation Model for Speech, Sound, Music, and Time-Aware Audio Reasoning

Understanding what’s happening in an audio clip is a deceptively hard problem. Transcribing spoken words is the easy part. A truly capable system also needs to recognize who is speaking, detect their emotional state, interpret background sounds, analyze musical content, and answer time-grounded questions like ‘what did the speaker say at the 2-minute mark?’. Tackling all of that required stitching together multiple specialized systems.

Tthe OpenMOSS team, MOSI.AI, and Shanghai Innovation Institute released MOSS-Audio: an open-source audio understanding model designed to unify all of those capabilities inside a single foundation model.

What MOSS-Audio Actually Does

MOSS-Audio supports speech understanding, environmental sound understanding, music understanding, audio captioning, time-aware QA, and complex reasoning over real-world audio. Its capability set breaks down into several distinct areas. Speech & Content Understanding accurately recognizes and transcribes spoken content, supporting both word-level and sentence-level timestamp alignment. Speaker, Emotion & Event Analysis identifies speaker characteristics, analyzes emotional states based on tone, timbre, and context, and detects key acoustic events within the audio. Scene & Sound Cue Extraction pulls meaningful signals from background sounds, environmental noise, and non-speech signals to infer scene context and atmosphere. Music Understanding analyzes musical style, emotional progression, and instrumentation. Audio Question Answering & Summarization handles questions and summaries across speech, podcasts, meetings, and interviews. Finally, Complex Reasoning performs multi-hop reasoning over audio content, powered by both chain-of-thought training and reinforcement learning.

In practical terms, a single MOSS-Audio model can do all of the above without switching between different specialized systems.

Four Model Variants

The team released four variants at launch: MOSS-Audio-4B-Instruct, MOSS-Audio-4B-Thinking, MOSS-Audio-8B-Instruct, and MOSS-Audio-8B-Thinking. The naming convention is worth understanding if you’re deciding which to use. The Instruct variants are optimized for direct instruction following, making them well-suited for production pipelines where you want predictable, structured outputs. The Thinking variants provide stronger chain-of-thought reasoning capabilities, better suited for tasks requiring multi-hop inference. The 4B models use Qwen3-4B as the LLM backbone, and the 8B models use Qwen3-8B, resulting in total model sizes of approximately 4.6B and 8.6B parameters respectively.

The Architecture: Three Components Working Together

MOSS-Audio follows a modular design comprising three components: an audio encoder, a modality adapter, and a large language model. Raw audio is first encoded by the MOSS-Audio-Encoder into continuous temporal representations at 12.5 Hz. Those representations are then projected into the language model’s embedding space through the adapter, and finally consumed by the LLM for auto-regressive text generation.

The research team trained the encoder from scratch rather than relying on off-the-shelf audio frontends. Their reasoning: a dedicated encoder delivers more robust speech representations, tighter temporal alignment, and better extensibility across acoustic domains.

Two architectural innovations inside MOSS-Audio are worth understanding in detail.

DeepStack Cross-Layer Feature Injection: A common weakness in audio models is that relying only on the encoder’s top-layer features tends to lose low-level acoustic information, things like prosody, transient events, and local time-frequency structure. MOSS-Audio addresses this with a DeepStack-inspired cross-layer injection module between the encoder and the language model: in addition to the encoder’s final-layer output, features from earlier and intermediate layers are selected, independently projected, and injected into the language model’s early layers. This preserves multi-granularity information ranging from low-level acoustic details to high-level semantic abstractions, helping the model retain rhythm, timbre, transients, and background structure that a single high-level representation cannot fully capture.

Time-Aware Representation: Time is a critical dimension in audio that text models aren’t naturally equipped to handle. MOSS-Audio addresses this through a time-marker insertion strategy during pretraining: explicit time tokens are inserted between audio frame representations at fixed time intervals to indicate temporal positions. This lets the model learn ‘what happened when’ within a unified text generation framework, naturally supporting timestamp ASR, event localization, time-based QA, and long-audio retrospection — without requiring a separate localization head or post-processing pipeline.

Benchmark Performance

The numbers are strong. On general audio understanding, MOSS-Audio-8B-Thinking achieves an average accuracy of 71.08 across four benchmarks — 77.33 on MMAU, 64.92 on MMAU-Pro, 66.53 on MMAR, and 75.52 on MMSU, outperforming majority of open-source models. That includes larger models: Step-Audio-R1 at 33B scores 70.67, and Qwen3-Omni-30B-A3B-Instruct at 30B scores 67.91. For further context, Kimi-Audio (7B) scores 61.14 and MiMo-Audio-7B scores 62.97 on the same average. The 4B Thinking variant scores 68.37, meaning the smaller model with chain-of-thought training beats all larger open-source instruct-only competitors.

On speech captioning, evaluated with an LLM-as-a-Judge methodology across 13 fine-grained dimensions including gender, age, accent, pitch, volume, speed, texture, clarity, fluency, emotion, tone, personality, and summary, MOSS-Audio-Instruct variants lead across 11 out of 13 dimensions, with MOSS-Audio-8B-Instruct achieving the best overall average score of 3.7252.

On automatic speech recognition (ASR) spanning 12 evaluation dimensions — including health condition, code-switching, dialect, singing, and non-speech scenarios — MOSS-Audio-8B-Instruct achieves the lowest overall CER (Character Error Rate) of 11.30 across all tested models.