Home › Audio Models › CSM (Conversational Speech Model)

CSM (Conversational Speech Model) - AI Audio Models Tool

Overview

CSM (Conversational Speech Model) is an open-source framework from SesameAILabs for generating conversational speech via discrete audio codes. The model converts text and short audio prompts into RVQ (residual vector quantization) audio codes using a Llama-based language backbone and a specialized audio decoder that emits Mimi audio codes. Those discrete codes can then be rendered by downstream vocoders or synthesis pipelines, enabling interactive, multi-turn speech generation that preserves conversational context. CSM is designed for research and integration into applications such as voice assistants, multimodal chatbots, accessibility tools, and synthetic dialogue systems. Because it outputs compact, quantized audio representations rather than raw waveforms, CSM can be combined with different vocoders, codec-aware pipelines, or streaming backends to control fidelity, latency, and bandwidth. The project is released under the Apache-2.0 license and emphasizes reproducible experiments, model interoperability, and extension points for fine-tuning language-conditioned audio behavior.

GitHub Statistics

  • Stars: 14,426
  • Forks: 1,462
  • Contributors: 11
  • License: Apache-2.0
  • Primary Language: Python
  • Last Updated: 2025-05-27T12:21:51Z

According to the GitHub repository, CSM has substantial community interest with 14,426 stars and 1,462 forks. The project lists 11 contributors and is actively maintained—the last commit was recorded on 2025-05-27. The Apache-2.0 license supports commercial and research reuse. High star and fork counts suggest broad adoption and experimentation; however, the relatively small core contributor count indicates community contributions are present but centralized. Active recent commits imply ongoing development and responsiveness to issues and feature requests.

Installation

Install via pip:

git clone https://github.com/SesameAILabs/csm.git
cd csm
pip install -r requirements.txt
pip install -e .

Key Features

  • Generates RVQ discrete audio codes from text and audio inputs
  • Uses a Llama backbone for language understanding and context
  • Specialized audio decoder that produces Mimi audio codes
  • Designed for interactive, multi-turn conversational speech synthesis
  • Apache-2.0 open-source license for research and commercial use

Community

The repository has strong visibility (14.4k stars, 1.46k forks) and active maintenance (last commit 2025-05-27). Community contributions exist (11 contributors) and users can engage via GitHub issues, pull requests, and the project's discussion spaces. Adoption appears strong among researchers and developers exploring discrete-code speech synthesis.

Last Refreshed: 2026-01-09

Key Information

  • Category: Audio Models
  • Type: AI Audio Models Tool
Visit Official Website

Related Tools

OpenVoice

OpenVoice is a versatile instant voice cloning framework that allows users to generate speech in multiple languages using only a short audio clip from a reference speaker. The tool provides granular control over voice styles, such as emotion, accent, rhythm, pauses, and intonation, and supports zero-shot cross-lingual voice cloning, enabling users to clone voices across different languages without needing training data for those languages.
Parler-TTS

A text-to-speech inference and training library for generating high-fidelity speech from text, offering an open-source solution for TTS applications.
SpeechBrain

An all-in-one open-source conversational AI toolkit based on PyTorch offering speech recognition, text-to-speech, speaker recognition, and more.
Whisper Large

A robust speech recognition model based on a Transformer architecture that supports multilingual transcription, speech translation, and language identification.
openai/whisper-large-v3-turbo

A finetuned, pruned version of Whisper large-v3 for automatic speech recognition and speech translation. This model reduces the number of decoding layers from 32 to 4 to achieve much faster inference, with only a minor quality trade-off. It supports 99 languages and integrates with Hugging Face Transformers for efficient transcription and translation.
OpenVoice V2

OpenVoice V2 is an advanced text-to-speech model that provides instant voice cloning with accurate tone color reproduction and flexible voice style control. It supports zero-shot cross-lingual synthesis in multiple languages and has improved audio quality over its previous version. Released under the MIT License, it is geared towards both research and commercial use.