diff --git a/README.md b/README.md index 9b4b115..3505d34 100644 --- a/README.md +++ b/README.md @@ -1,92 +1,87 @@ # Qwen3-TTS Streaming -Streaming inference implementation for [Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS) that the official repo doesn't provide. +Real-time streaming audio generation for [Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS). -The official team mentions "Extreme Low-Latency Streaming Generation" in their paper and marketing, but the actual streaming code was never released - they point users to vLLM-Omni, which still doesn't support online serving. +## Features -This fork adds real streaming generation directly to the `qwen-tts` package. - -## What's Added - -- `stream_generate_pcm()` - real-time PCM audio streaming +From [dffdeeq/Qwen3-TTS-streaming](https://github.com/dffdeeq/Qwen3-TTS-streaming): - `stream_generate_voice_clone()` - streaming with voice cloning +- `stream_generate_pcm()` - real-time PCM audio streaming +- `torch.compile` + CUDA graphs optimization +- Crossfade overlap for seamless chunk transitions -## Benchmark (RTX 5090) +Added in this fork: +- **Two-phase streaming** - faster first-chunk latency -### Non-streaming (full inference) +## Two-Phase Streaming -image +Standard streaming with Qwen's TTS library waits for `emit_every_frames` (e.g., 12) before emitting the first audio. Two-phase uses aggressive settings for the first chunk to improve latency, then switches to stable settings. +``` +┌─────────────────────────────────────────────────────────────────┐ +│ PHASE 1 (First N frames) │ PHASE 2 (Rest of audio) │ +│ - emit_every = 5 (fast) │ - emit_every = 12 (stable) │ +│ - decode_window = 48 │ - decode_window = 80 │ +│ - optimized = OFF │ - optimized = ON │ +│ → FAST first chunk │ → QUALITY for rest │ +└─────────────────────────────────────────────────────────────────┘ +``` -### Streaming +### Benchmarks -image +| Test | Method | emit | 1st Chunk | 1st Spdup | Total | Tot Spdup | RTF | +|------|--------|------|-----------|-----------|-------|-----------|-----| +| 2 | Baseline (no opt) | 12 | 570ms | 1.00x | 3.16s | 1.00x | 0.56 | +| 3 | Optimized | 12 | 389ms | 1.47x | 2.37s | 1.34x | 0.37 | +| 4 | Optimized_2 (stable) | 12 | 382ms | 1.49x | 2.27s | 1.39x | 0.36 | +| 5 | **Two-phase (5→12)** | 5→12 | **208ms** | **2.75x** | 2.58s | 1.23x | 0.39 | +User hears audio **362ms earlier** vs baseline, **174ms earlier** vs only optimized. + +**First-chunk latency improvement:** +- vs Baseline: **2.75x faster** (570ms → 208ms, saves 362ms) +- vs Optimized: **1.87x faster** (389ms → 208ms, saves 181ms) +- vs Optimized_2: **1.84x faster** (382ms → 208ms, saves 174ms) ## Usage -See examples/ -- [test_streaming_optimized.py](https://github.com/dffdeeq/Qwen3-TTS-streaming/blob/main/examples/test_streaming_optimized.py) -- [test_optimized_no_streaming.py](https://github.com/dffdeeq/Qwen3-TTS-streaming/blob/main/examples/test_optimized_no_streaming.py) - -## Installation (python 3.12) - -> Note: torch versions differ between Linux/Windows due to available flash_attn prebuilt wheels. - -### 1. Install SOX - -**Linux:** -```bash -sudo apt install sox libsox-fmt-all -``` - -**Windows:** -```bash -# Download from https://sourceforge.net/projects/sox/ and add to PATH !! -``` - -### 2. Create environment -```bash -conda create -n qwen3-tts python=3.12 -y -conda activate qwen3-tts -``` - -### 3. Install dependencies - -**Linux:** -```bash -pip install torch==2.9.1 torchaudio==2.9.1 --index-url https://download.pytorch.org/whl/cu130 -pip install https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.6.8/flash_attn-2.8.3%2Bcu130torch2.9-cp312-cp312-linux_x86_64.whl -``` - -**Windows:** -```bash -pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu130 -pip install https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.7.12/flash_attn-2.8.3%2Bcu130torch2.10-cp312-cp312-win_amd64.whl -pip install -U "triton-windows<3.7" -``` - -### 4. Install package -```bash -git clone https://github.com/dffdeeq/Qwen3-TTS-streaming.git -cd Qwen3-TTS-streaming -pip install -e . +```python +for chunk, sr in model.stream_generate_voice_clone( + text="Hello!", + language="en", + voice_clone_prompt=prompt, + # Phase 2 settings + emit_every_frames=12, + decode_window_frames=80, + # Phase 1 settings (two-phase) + first_chunk_emit_every=5, + first_chunk_decode_window=48, + first_chunk_frames=48, +): + play_audio(chunk, sr) ``` ## Parameters | Parameter | Default | Description | |-----------|---------|-------------| -| `emit_every_frames` | 4 | Emit audio every N frames (~0.33s at 12Hz) | +| `emit_every_frames` | 8 | Emit audio every N frames | | `decode_window_frames` | 80 | Decoder context window | +| `overlap_samples` | 512 | Crossfade overlap between chunks | +| `first_chunk_emit_every` | 0 | Phase 1 emit interval (0 = disabled) | +| `first_chunk_decode_window` | 48 | Phase 1 decode window | +| `first_chunk_frames` | 48 | Switch to phase 2 after N frames | -## Why This Exists +## Installation -From official Qwen3-TTS README: -> Now only offline inference is supported. Online serving will be supported later. - -This fork provides streaming now, without waiting for vLLM-Omni updates. +```bash +sudo apt install sox +pip install torch torchaudio flash-attn +pip install -e . +``` --- -Based on [QwenLM/Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS) +Based on: +- [QwenLM/Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS) +- [dffdeeq/Qwen3-TTS-streaming](https://github.com/dffdeeq/Qwen3-TTS-streaming) diff --git a/qwen_tts/README.md b/qwen_tts/README.md new file mode 100644 index 0000000..3505d34 --- /dev/null +++ b/qwen_tts/README.md @@ -0,0 +1,87 @@ +# Qwen3-TTS Streaming + +Real-time streaming audio generation for [Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS). + +## Features + +From [dffdeeq/Qwen3-TTS-streaming](https://github.com/dffdeeq/Qwen3-TTS-streaming): +- `stream_generate_voice_clone()` - streaming with voice cloning +- `stream_generate_pcm()` - real-time PCM audio streaming +- `torch.compile` + CUDA graphs optimization +- Crossfade overlap for seamless chunk transitions + +Added in this fork: +- **Two-phase streaming** - faster first-chunk latency + +## Two-Phase Streaming + +Standard streaming with Qwen's TTS library waits for `emit_every_frames` (e.g., 12) before emitting the first audio. Two-phase uses aggressive settings for the first chunk to improve latency, then switches to stable settings. + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ PHASE 1 (First N frames) │ PHASE 2 (Rest of audio) │ +│ - emit_every = 5 (fast) │ - emit_every = 12 (stable) │ +│ - decode_window = 48 │ - decode_window = 80 │ +│ - optimized = OFF │ - optimized = ON │ +│ → FAST first chunk │ → QUALITY for rest │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### Benchmarks + +| Test | Method | emit | 1st Chunk | 1st Spdup | Total | Tot Spdup | RTF | +|------|--------|------|-----------|-----------|-------|-----------|-----| +| 2 | Baseline (no opt) | 12 | 570ms | 1.00x | 3.16s | 1.00x | 0.56 | +| 3 | Optimized | 12 | 389ms | 1.47x | 2.37s | 1.34x | 0.37 | +| 4 | Optimized_2 (stable) | 12 | 382ms | 1.49x | 2.27s | 1.39x | 0.36 | +| 5 | **Two-phase (5→12)** | 5→12 | **208ms** | **2.75x** | 2.58s | 1.23x | 0.39 | + +User hears audio **362ms earlier** vs baseline, **174ms earlier** vs only optimized. + +**First-chunk latency improvement:** +- vs Baseline: **2.75x faster** (570ms → 208ms, saves 362ms) +- vs Optimized: **1.87x faster** (389ms → 208ms, saves 181ms) +- vs Optimized_2: **1.84x faster** (382ms → 208ms, saves 174ms) + +## Usage + +```python +for chunk, sr in model.stream_generate_voice_clone( + text="Hello!", + language="en", + voice_clone_prompt=prompt, + # Phase 2 settings + emit_every_frames=12, + decode_window_frames=80, + # Phase 1 settings (two-phase) + first_chunk_emit_every=5, + first_chunk_decode_window=48, + first_chunk_frames=48, +): + play_audio(chunk, sr) +``` + +## Parameters + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `emit_every_frames` | 8 | Emit audio every N frames | +| `decode_window_frames` | 80 | Decoder context window | +| `overlap_samples` | 512 | Crossfade overlap between chunks | +| `first_chunk_emit_every` | 0 | Phase 1 emit interval (0 = disabled) | +| `first_chunk_decode_window` | 48 | Phase 1 decode window | +| `first_chunk_frames` | 48 | Switch to phase 2 after N frames | + +## Installation + +```bash +sudo apt install sox +pip install torch torchaudio flash-attn +pip install -e . +``` + +--- + +Based on: +- [QwenLM/Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS) +- [dffdeeq/Qwen3-TTS-streaming](https://github.com/dffdeeq/Qwen3-TTS-streaming) diff --git a/qwen_tts/core/models/modeling_qwen3_tts.py b/qwen_tts/core/models/modeling_qwen3_tts.py index bd8f02a..7a632f7 100644 --- a/qwen_tts/core/models/modeling_qwen3_tts.py +++ b/qwen_tts/core/models/modeling_qwen3_tts.py @@ -2605,6 +2605,10 @@ class Qwen3TTSForConditionalGeneration(Qwen3TTSPreTrainedModel, GenerationMixin) max_frames: int = 10000, # Optimization flags use_optimized_decode: bool = True, + # Two-phase streaming: aggressive first chunk + first_chunk_emit_every: int = 0, # 0 = disabled, use emit_every_frames throughout + first_chunk_decode_window: int = 48, + first_chunk_frames: int = 48, # Switch to stable after this many frames ) -> Generator[tuple[np.ndarray, int], None, None]: """ Stream audio generation, yielding PCM chunks as they are generated. @@ -2627,6 +2631,9 @@ class Qwen3TTSForConditionalGeneration(Qwen3TTSPreTrainedModel, GenerationMixin) overlap_samples: Overlap samples for crossfade between chunks max_frames: Maximum number of codec frames to generate use_optimized_decode: Use CUDA graph optimized decode when available (default True) + first_chunk_emit_every: Emit interval for first chunk phase (0 = disabled, use emit_every_frames) + first_chunk_decode_window: Decode window size for first chunk phase + first_chunk_frames: Switch to stable settings after this many frames Yields: tuple[np.ndarray, int]: (pcm_chunk as float32 array, sample_rate) @@ -2751,22 +2758,36 @@ class Qwen3TTSForConditionalGeneration(Qwen3TTSPreTrainedModel, GenerationMixin) token = torch.argmax(step_logits, dim=-1) frames_since_emit += 1 - if frames_since_emit < emit_every_frames: + + # Two-phase streaming: determine current phase settings + total_frames_generated = len(codes_buffer) + if first_chunk_emit_every > 0 and total_frames_generated < first_chunk_frames: + # Phase 1: Aggressive settings for first chunk (lower latency) + current_emit_every = first_chunk_emit_every + current_decode_window = first_chunk_decode_window + current_use_optimized = False # Non-optimized allows flexible window size + else: + # Phase 2: Stable settings (better quality) + current_emit_every = emit_every_frames + current_decode_window = decode_window_frames + current_use_optimized = use_optimized_decode + + if frames_since_emit < current_emit_every: continue frames_since_emit = 0 # Decode window of codec frames to PCM - start = max(0, len(codes_buffer) - decode_window_frames) + start = max(0, len(codes_buffer) - current_decode_window) window_codes = torch.stack(codes_buffer[start:], dim=0) # [T, num_code_groups] # Add ref_code as context prefix for stable decoder context from the start window, _ = _add_ref_code_context( - window_codes, ref_code_context, ref_code_frames, decode_window_frames + window_codes, ref_code_context, ref_code_frames, current_decode_window ) # Use optimized decode path when available # Pass pad_to_size to ensure fixed tensor size for torch.compile - if use_optimized_decode and hasattr(self.speech_tokenizer, 'decode_streaming'): + if current_use_optimized and hasattr(self.speech_tokenizer, 'decode_streaming'): wavs, sr = self.speech_tokenizer.decode_streaming( window.to(self.talker.device), use_optimized=True, @@ -2781,7 +2802,7 @@ class Qwen3TTSForConditionalGeneration(Qwen3TTSPreTrainedModel, GenerationMixin) # Extract only new samples (tail of decoded window) # Use fixed upsample rate to avoid floating-point drift samples_per_frame = self.speech_tokenizer.get_decode_upsample_rate() - step_samples = samples_per_frame * emit_every_frames + step_samples = samples_per_frame * current_emit_every chunk = wav[-step_samples:] if step_samples > 0 else wav # Crossfade with previous chunk tail for smooth transition diff --git a/qwen_tts/inference/qwen3_tts_model.py b/qwen_tts/inference/qwen3_tts_model.py index d1a4962..75b6d1b 100644 --- a/qwen_tts/inference/qwen3_tts_model.py +++ b/qwen_tts/inference/qwen3_tts_model.py @@ -700,6 +700,10 @@ class Qwen3TTSModel: max_frames: int = 10000, # Optimization use_optimized_decode: bool = True, + # Two-phase streaming: aggressive first chunk + first_chunk_emit_every: int = 0, # 0 = disabled, use emit_every_frames throughout + first_chunk_decode_window: int = 48, + first_chunk_frames: int = 48, # Switch to stable after this many frames **kwargs, ) -> Generator[Tuple[np.ndarray, int], None, None]: """ @@ -721,6 +725,9 @@ class Qwen3TTSModel: max_frames: Maximum codec frames to generate. use_optimized_decode: Use CUDA graph optimized decode when available (default True). Call enable_streaming_optimizations() first for best performance. + first_chunk_emit_every: Emit interval for first chunk phase (0 = disabled). + first_chunk_decode_window: Decode window size for first chunk phase. + first_chunk_frames: Switch to stable settings after this many frames. **kwargs: Generation parameters (do_sample, top_k, top_p, temperature, etc.) Yields: @@ -798,6 +805,9 @@ class Qwen3TTSModel: overlap_samples=overlap_samples, max_frames=max_frames, use_optimized_decode=use_optimized_decode, + first_chunk_emit_every=first_chunk_emit_every, + first_chunk_decode_window=first_chunk_decode_window, + first_chunk_frames=first_chunk_frames, **gen_kwargs, ): yield chunk, sr