Nighthawk/Qwen3-TTS-streaming
1
0
Fork 0
mirror of https://github.com/Nighthawk42/Qwen3-TTS-streaming.git synced 2026-08-30 08:52:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1 from rekuenkdr/feature/two-phase-emit

Browse Source 
feat: add two-phase streaming for reduced first-chunk latency
This commit is contained in:
rekuenkdr
2026-02-02 20:29:27 +01:00
committed by GitHub
parent 01b51f0cb1 7bf2c43a2e
commit 92aaeac93c
4 changed files with 183 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+60 -65
View File
@@ -1,92 +1,87 @@
# Qwen3-TTS Streaming
Streaming inference implementation for [Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS) that the official repo doesn't provide.
Real-time streaming audio generation for [Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS).
The official team mentions "Extreme Low-Latency Streaming Generation" in their paper and marketing, but the actual streaming code was never released - they point users to vLLM-Omni, which still doesn't support online serving.
## Features
This fork adds real streaming generation directly to the `qwen-tts` package.
## What's Added
- `stream_generate_pcm()` - real-time PCM audio streaming
From [dffdeeq/Qwen3-TTS-streaming](https://github.com/dffdeeq/Qwen3-TTS-streaming):
- `stream_generate_voice_clone()` - streaming with voice cloning
- `stream_generate_pcm()` - real-time PCM audio streaming
- `torch.compile` + CUDA graphs optimization
- Crossfade overlap for seamless chunk transitions
## Benchmark (RTX 5090)
Added in this fork:
- **Two-phase streaming** - faster first-chunk latency
### Non-streaming (full inference)
## Two-Phase Streaming
<img width="575" height="136" alt="image" src="https://github.com/user-attachments/assets/e0744b19-d31b-44df-9f61-5be56f29056b" />
Standard streaming with Qwen's TTS library waits for `emit_every_frames` (e.g., 12) before emitting the first audio. Two-phase uses aggressive settings for the first chunk to improve latency, then switches to stable settings.
```
┌─────────────────────────────────────────────────────────────────┐
│ PHASE 1 (First N frames) │ PHASE 2 (Rest of audio) │
│ - emit_every = 5 (fast) │ - emit_every = 12 (stable) │
│ - decode_window = 48 │ - decode_window = 80 │
│ - optimized = OFF │ - optimized = ON │
│ → FAST first chunk │ → QUALITY for rest │
└─────────────────────────────────────────────────────────────────┘
```
### Streaming
### Benchmarks
<img width="738" height="130" alt="image" src="https://github.com/user-attachments/assets/5694aa47-f2db-4b97-ae41-4d834838f8d8" />
| Test | Method | emit | 1st Chunk | 1st Spdup | Total | Tot Spdup | RTF |
|------|--------|------|-----------|-----------|-------|-----------|-----|
| 2 | Baseline (no opt) | 12 | 570ms | 1.00x | 3.16s | 1.00x | 0.56 |
| 3 | Optimized | 12 | 389ms | 1.47x | 2.37s | 1.34x | 0.37 |
| 4 | Optimized_2 (stable) | 12 | 382ms | 1.49x | 2.27s | 1.39x | 0.36 |
| 5 | **Two-phase (5→12)** | 5→12 | **208ms** | **2.75x** | 2.58s | 1.23x | 0.39 |
User hears audio **362ms earlier** vs baseline, **174ms earlier** vs only optimized.
**First-chunk latency improvement:**
- vs Baseline: **2.75x faster** (570ms → 208ms, saves 362ms)
- vs Optimized: **1.87x faster** (389ms → 208ms, saves 181ms)
- vs Optimized_2: **1.84x faster** (382ms → 208ms, saves 174ms)
## Usage
See examples/
- [test_streaming_optimized.py](https://github.com/dffdeeq/Qwen3-TTS-streaming/blob/main/examples/test_streaming_optimized.py)
- [test_optimized_no_streaming.py](https://github.com/dffdeeq/Qwen3-TTS-streaming/blob/main/examples/test_optimized_no_streaming.py)
## Installation (python 3.12)
> Note: torch versions differ between Linux/Windows due to available flash_attn prebuilt wheels.
### 1. Install SOX
**Linux:**
```bash
sudo apt install sox libsox-fmt-all
```
**Windows:**
```bash
# Download from https://sourceforge.net/projects/sox/ and add to PATH !!
```
### 2. Create environment
```bash
conda create -n qwen3-tts python=3.12 -y
conda activate qwen3-tts
```
### 3. Install dependencies
**Linux:**
```bash
pip install torch==2.9.1 torchaudio==2.9.1 --index-url https://download.pytorch.org/whl/cu130
pip install https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.6.8/flash_attn-2.8.3%2Bcu130torch2.9-cp312-cp312-linux_x86_64.whl
```
**Windows:**
```bash
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu130
pip install https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.7.12/flash_attn-2.8.3%2Bcu130torch2.10-cp312-cp312-win_amd64.whl
pip install -U "triton-windows<3.7"
```
### 4. Install package
```bash
git clone https://github.com/dffdeeq/Qwen3-TTS-streaming.git
cd Qwen3-TTS-streaming
pip install -e .
```python
for chunk, sr in model.stream_generate_voice_clone(
text="Hello!",
language="en",
voice_clone_prompt=prompt,
# Phase 2 settings
emit_every_frames=12,
decode_window_frames=80,
# Phase 1 settings (two-phase)
first_chunk_emit_every=5,
first_chunk_decode_window=48,
first_chunk_frames=48,
):
play_audio(chunk, sr)
```
## Parameters
| Parameter | Default | Description |
|-----------|---------|-------------|
| `emit_every_frames` | 4 | Emit audio every N frames (~0.33s at 12Hz) |
| `emit_every_frames` | 8 | Emit audio every N frames |
| `decode_window_frames` | 80 | Decoder context window |
| `overlap_samples` | 512 | Crossfade overlap between chunks |
| `first_chunk_emit_every` | 0 | Phase 1 emit interval (0 = disabled) |
| `first_chunk_decode_window` | 48 | Phase 1 decode window |
| `first_chunk_frames` | 48 | Switch to phase 2 after N frames |
## Why This Exists
## Installation
From official Qwen3-TTS README:
> Now only offline inference is supported. Online serving will be supported later.
This fork provides streaming now, without waiting for vLLM-Omni updates.
```bash
sudo apt install sox
pip install torch torchaudio flash-attn
pip install -e .
```
---
Based on [QwenLM/Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS)
Based on:
- [QwenLM/Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS)
- [dffdeeq/Qwen3-TTS-streaming](https://github.com/dffdeeq/Qwen3-TTS-streaming)
qwen_tts/README.md
+87
View File
@@ -0,0 +1,87 @@
# Qwen3-TTS Streaming
Real-time streaming audio generation for [Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS).
## Features
From [dffdeeq/Qwen3-TTS-streaming](https://github.com/dffdeeq/Qwen3-TTS-streaming):
- `stream_generate_voice_clone()` - streaming with voice cloning
- `stream_generate_pcm()` - real-time PCM audio streaming
- `torch.compile` + CUDA graphs optimization
- Crossfade overlap for seamless chunk transitions
Added in this fork:
- **Two-phase streaming** - faster first-chunk latency
## Two-Phase Streaming
Standard streaming with Qwen's TTS library waits for `emit_every_frames` (e.g., 12) before emitting the first audio. Two-phase uses aggressive settings for the first chunk to improve latency, then switches to stable settings.
```
┌─────────────────────────────────────────────────────────────────┐
│ PHASE 1 (First N frames) │ PHASE 2 (Rest of audio) │
│ - emit_every = 5 (fast) │ - emit_every = 12 (stable) │
│ - decode_window = 48 │ - decode_window = 80 │
│ - optimized = OFF │ - optimized = ON │
│ → FAST first chunk │ → QUALITY for rest │
└─────────────────────────────────────────────────────────────────┘
```
### Benchmarks
| Test | Method | emit | 1st Chunk | 1st Spdup | Total | Tot Spdup | RTF |
|------|--------|------|-----------|-----------|-------|-----------|-----|
| 2 | Baseline (no opt) | 12 | 570ms | 1.00x | 3.16s | 1.00x | 0.56 |
| 3 | Optimized | 12 | 389ms | 1.47x | 2.37s | 1.34x | 0.37 |
| 4 | Optimized_2 (stable) | 12 | 382ms | 1.49x | 2.27s | 1.39x | 0.36 |
| 5 | **Two-phase (5→12)** | 5→12 | **208ms** | **2.75x** | 2.58s | 1.23x | 0.39 |
User hears audio **362ms earlier** vs baseline, **174ms earlier** vs only optimized.
**First-chunk latency improvement:**
- vs Baseline: **2.75x faster** (570ms → 208ms, saves 362ms)
- vs Optimized: **1.87x faster** (389ms → 208ms, saves 181ms)
- vs Optimized_2: **1.84x faster** (382ms → 208ms, saves 174ms)
## Usage
```python
for chunk, sr in model.stream_generate_voice_clone(
text="Hello!",
language="en",
voice_clone_prompt=prompt,
# Phase 2 settings
emit_every_frames=12,
decode_window_frames=80,
# Phase 1 settings (two-phase)
first_chunk_emit_every=5,
first_chunk_decode_window=48,
first_chunk_frames=48,
):
play_audio(chunk, sr)
```
## Parameters
| Parameter | Default | Description |
|-----------|---------|-------------|
| `emit_every_frames` | 8 | Emit audio every N frames |
| `decode_window_frames` | 80 | Decoder context window |
| `overlap_samples` | 512 | Crossfade overlap between chunks |
| `first_chunk_emit_every` | 0 | Phase 1 emit interval (0 = disabled) |
| `first_chunk_decode_window` | 48 | Phase 1 decode window |
| `first_chunk_frames` | 48 | Switch to phase 2 after N frames |
## Installation
```bash
sudo apt install sox
pip install torch torchaudio flash-attn
pip install -e .
```
---
Based on:
- [QwenLM/Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS)
- [dffdeeq/Qwen3-TTS-streaming](https://github.com/dffdeeq/Qwen3-TTS-streaming)
qwen_tts/core/models/modeling_qwen3_tts.py
+26 -5
View File
@@ -2605,6 +2605,10 @@ class Qwen3TTSForConditionalGeneration(Qwen3TTSPreTrainedModel, GenerationMixin)
max_frames: int = 10000,
# Optimization flags
use_optimized_decode: bool = True,
# Two-phase streaming: aggressive first chunk
first_chunk_emit_every: int = 0, # 0 = disabled, use emit_every_frames throughout
first_chunk_decode_window: int = 48,
first_chunk_frames: int = 48, # Switch to stable after this many frames
) -> Generator[tuple[np.ndarray, int], None, None]:
"""
Stream audio generation, yielding PCM chunks as they are generated.
@@ -2627,6 +2631,9 @@ class Qwen3TTSForConditionalGeneration(Qwen3TTSPreTrainedModel, GenerationMixin)
overlap_samples: Overlap samples for crossfade between chunks
max_frames: Maximum number of codec frames to generate
use_optimized_decode: Use CUDA graph optimized decode when available (default True)
first_chunk_emit_every: Emit interval for first chunk phase (0 = disabled, use emit_every_frames)
first_chunk_decode_window: Decode window size for first chunk phase
first_chunk_frames: Switch to stable settings after this many frames
Yields:
tuple[np.ndarray, int]: (pcm_chunk as float32 array, sample_rate)
@@ -2751,22 +2758,36 @@ class Qwen3TTSForConditionalGeneration(Qwen3TTSPreTrainedModel, GenerationMixin)
token = torch.argmax(step_logits, dim=-1)
frames_since_emit += 1
if frames_since_emit < emit_every_frames:
# Two-phase streaming: determine current phase settings
total_frames_generated = len(codes_buffer)
if first_chunk_emit_every > 0 and total_frames_generated < first_chunk_frames:
# Phase 1: Aggressive settings for first chunk (lower latency)
current_emit_every = first_chunk_emit_every
current_decode_window = first_chunk_decode_window
current_use_optimized = False # Non-optimized allows flexible window size
else:
# Phase 2: Stable settings (better quality)
current_emit_every = emit_every_frames
current_decode_window = decode_window_frames
current_use_optimized = use_optimized_decode
if frames_since_emit < current_emit_every:
continue
frames_since_emit = 0
# Decode window of codec frames to PCM
start = max(0, len(codes_buffer) - decode_window_frames)
start = max(0, len(codes_buffer) - current_decode_window)
window_codes = torch.stack(codes_buffer[start:], dim=0) # [T, num_code_groups]
# Add ref_code as context prefix for stable decoder context from the start
window, _ = _add_ref_code_context(
window_codes, ref_code_context, ref_code_frames, decode_window_frames
window_codes, ref_code_context, ref_code_frames, current_decode_window
)
# Use optimized decode path when available
# Pass pad_to_size to ensure fixed tensor size for torch.compile
if use_optimized_decode and hasattr(self.speech_tokenizer, 'decode_streaming'):
if current_use_optimized and hasattr(self.speech_tokenizer, 'decode_streaming'):
wavs, sr = self.speech_tokenizer.decode_streaming(
window.to(self.talker.device),
use_optimized=True,
@@ -2781,7 +2802,7 @@ class Qwen3TTSForConditionalGeneration(Qwen3TTSPreTrainedModel, GenerationMixin)
# Extract only new samples (tail of decoded window)
# Use fixed upsample rate to avoid floating-point drift
samples_per_frame = self.speech_tokenizer.get_decode_upsample_rate()
step_samples = samples_per_frame * emit_every_frames
step_samples = samples_per_frame * current_emit_every
chunk = wav[-step_samples:] if step_samples > 0 else wav
# Crossfade with previous chunk tail for smooth transition
qwen_tts/inference/qwen3_tts_model.py
+10
View File
@@ -700,6 +700,10 @@ class Qwen3TTSModel:
max_frames: int = 10000,
# Optimization
use_optimized_decode: bool = True,
# Two-phase streaming: aggressive first chunk
first_chunk_emit_every: int = 0, # 0 = disabled, use emit_every_frames throughout
first_chunk_decode_window: int = 48,
first_chunk_frames: int = 48, # Switch to stable after this many frames
**kwargs,
) -> Generator[Tuple[np.ndarray, int], None, None]:
"""
@@ -721,6 +725,9 @@ class Qwen3TTSModel:
max_frames: Maximum codec frames to generate.
use_optimized_decode: Use CUDA graph optimized decode when available (default True).
Call enable_streaming_optimizations() first for best performance.
first_chunk_emit_every: Emit interval for first chunk phase (0 = disabled).
first_chunk_decode_window: Decode window size for first chunk phase.
first_chunk_frames: Switch to stable settings after this many frames.
**kwargs: Generation parameters (do_sample, top_k, top_p, temperature, etc.)
Yields:
@@ -798,6 +805,9 @@ class Qwen3TTSModel:
overlap_samples=overlap_samples,
max_frames=max_frames,
use_optimized_decode=use_optimized_decode,
first_chunk_emit_every=first_chunk_emit_every,
first_chunk_decode_window=first_chunk_decode_window,
first_chunk_frames=first_chunk_frames,
**gen_kwargs,
):
yield chunk, sr