The unbounded repetition penalty was causing sentence repetition by
progressively starving the 2048-token codec vocabulary. Streaming
wrappers now default to repetition_penalty=1.0 (disabled), passed as
an explicit parameter that bypasses _merge_generate_kwargs' 1.05
default. Non-streaming generate_voice_clone() is unaffected.
Also fix decode_padded to pad with -1 instead of 0 (a valid codebook
index), then clamp to >= 0 before decoding, avoiding silent corruption
of the first frames in windowed streaming decode.
Port two upstream bug fixes:
1. ConvTranspose padding (5f8581d): the chained assignment
`self.right_pad = pad = self.left_pad` destroyed the original pad
value, causing symmetric trimming instead of right-only trimming.
This lost 20 extra samples per decode pass across the 4 decoder
blocks. Fix: set left_pad=0, right_pad=int(pad).
2. Padding value / audio_lengths (6cafe55): batch padding used value 0,
but code index 0 is a valid codebook entry. audio_lengths computed
with `> 0` then incorrectly treated real code-0 tokens as padding,
truncating audio. Fix: pad with -1, compute lengths with `> -1`,
then clamp codes to min=0 before decoding.
Upstream commits:
- https://github.com/QwenLM/Qwen3-TTS/commit/5f8581d05e70d2a11c6f0787284eede65f044d98
- https://github.com/QwenLM/Qwen3-TTS/commit/6cafe5582caea83df269c36b1ce62d953a9cc66b
Reverts a change we applied from open upstream PR #178 (commit f83f184),
which removed the sub-codebook embedding loop arguing the talker shouldn't
see levels 1-15 during training. Investigation of the inference code
(modeling_qwen3_tts.py:1894-1920) shows the sub-talker generates levels
1-15 before the talker's next step, and all 16 levels are summed into
the input. Omitting them from training creates a train/inference mismatch.
The HF ForCausalLMLoss already shifts labels internally, but sft_12hz.py
was also manually shifting before passing to model.talker(), causing
double-shifting and progressively faster audio with more epochs.
Additionally, the sub-talker forward_finetune used ForCausalLMLoss on
already-aligned logits/labels, adding another unwanted shift. Replaced
with direct cross_entropy.
See: https://github.com/QwenLM/Qwen3-TTS/issues/179#issuecomment-3870059313
Restructure the batch streaming loop and flush into a 3-phase approach:
1) collect decode windows for all active items, 2) perform a single
batched GPU decode call, 3) per-item post-processing (crossfade, trim).
Add decode_streaming_batch() to Qwen3TTSTokenizer to support B>1 batch
decoding. Remove examples/test_batch_streaming.py test script.
Document the new batch_stream_generate_voice_clone() method including
a usage example showing voice prompt broadcasting, per-item chunk
accumulation, and WAV output. Add entry to the "Added in this fork" list.
Introduce batch streaming methods that process multiple text inputs in a
single batched pass through the transformer, enabling parallel audio
generation with shared KV cache and lockstep frame advancement.
Core model layer (modeling_qwen3_tts.py):
- Add `batch_stream_generate_pcm()` to Qwen3TTSForConditionalGeneration
- Batched prefill and single-step decode with shared KV cache across items
- Per-item state management for codec buffers, crossfade tails, repetition
penalty tracking, ref_code ICL contexts, and independent EOS detection
- Two-phase streaming support: aggressive first-chunk emission followed by
stable-phase parameters for balancing latency vs quality
- Windowed decode with per-item Hann fade-in/out and overlap crossfade
- Flush pass to decode remaining frames per item after generation completes
High-level API (qwen3_tts_model.py):
- Add `batch_stream_generate_voice_clone()` to Qwen3TTSModel
- Handles broadcasting of language and voice_clone_prompt to batch size
- Input validation, tokenization, and ref_ids construction per item
- Filters and merges generation kwargs before delegating to core method
Example script (examples/test_batch_streaming.py):
- Demonstrates batch streaming with 3 texts using a shared voice prompt
- Benchmarks batch vs sequential single-item streaming for comparison
- Saves per-item WAV outputs with timing and size reporting
fix: add repetition penalty to streaming and sync upstream finetuning
Streaming inference (voice clone) - repetition penalty:
Without repetition penalty, the model can fall into a degenerate state where it keeps sampling the same codec tokens over and over. This manifests as:
Looping audio: the same syllable or sound fragment repeats endlessly
Extremely long generation: instead of reaching EOS in ~200-500 frames, it runs for thousands of frames (up to max_frames=10000)
Apparent "slowness": a response that should take ~1s of audio takes 10-30s to generate
The fix works by tracking previously generated first-codebook token IDs and penalizing them before sampling:
Tokens with positive logits get divided by repetition_penalty (lowering their probability)
Tokens with negative logits get multiplied by it (pushing them further down)
This nudges the model away from re-selecting the same tokens, so it progresses through the text naturally and reaches EOS in a reasonable number of steps rather than looping. Default is 1.0 (disabled) and is exposed through the supported_params whitelist in
stream_generate_voice_clone() so it can be set via generate_config or user kwargs.
Upstream sync (QwenLM/Qwen3-TTS):
Bump version 0.0.4 -> 0.1.1 to match upstream release.
finetuning/sft_12hz.py: weight sub-talker loss by 0.3 factor to prevent the code predictor gradient from dominating the main talker loss during SFT.
finetuning/sft_12hz.py: remove sub-codebook embedding accumulation loop (codec groups 1-15) from input embeddings, unnecessary and harmful for finetuning convergence (upstream PR Resolve each training epoch resulting in progressively faster output QwenLM/Qwen3-TTS#178).
finetuning/README.md: update recommended hyperparameters to batch_size=32, lr=2e-6, num_epochs=10 for more stable training.
Streaming inference (voice clone) - repetition penalty:
Without repetition penalty, the model can fall into a degenerate state
where it keeps sampling the same codec tokens over and over. This
manifests as:
- Looping audio: the same syllable or sound fragment repeats endlessly
- Extremely long generation: instead of reaching EOS in ~200-500 frames,
it runs for thousands of frames (up to max_frames=10000)
- Apparent "slowness": a response that should take ~1s of audio takes
10-30s to generate
The fix works by tracking previously generated first-codebook token IDs
and penalizing them before sampling:
- Tokens with positive logits get divided by repetition_penalty (lowering
their probability)
- Tokens with negative logits get multiplied by it (pushing them further
down)
This nudges the model away from re-selecting the same tokens, so it
progresses through the text naturally and reaches EOS in a reasonable
number of steps rather than looping. Default is 1.0 (disabled) and is
exposed through the supported_params whitelist in
stream_generate_voice_clone() so it can be set via generate_config or
user kwargs.
Upstream sync (QwenLM/Qwen3-TTS):
- Bump version 0.0.4 -> 0.1.1 to match upstream release.
- finetuning/sft_12hz.py: weight sub-talker loss by 0.3 factor to
prevent the code predictor gradient from dominating the main talker
loss during SFT.
- finetuning/sft_12hz.py: remove sub-codebook embedding accumulation
loop (codec groups 1-15) from input embeddings, unnecessary and
harmful for finetuning convergence (upstream PR #178).
- finetuning/README.md: update recommended hyperparameters to
batch_size=32, lr=2e-6, num_epochs=10 for more stable training.
fix: handle multiple EOS tokens for generation termination
Add support for multiple EOS tokens that can terminate TTS generation, rather than just checking for a single codec EOS token. This handles different EOS tokens the model might emit (codec EOS, TTS special tokens, endoftext, etc.) in order to avoid the issues with the 0.6B model.
Allow users to set any overlap_samples value including 0 to disable
crossfade blending entirely. Renamed MIN_BLEND_SAMPLES to
DEFAULT_BLEND_SAMPLES and removed the enforced minimum.
- Replace linear crossfade with Hann window for smoother transitions
- Add MIN_BLEND_SAMPLES (512) as floor for blend region
- Add Hann fade-in on first chunk to prevent startup pop
- Add Hann fade-out on final chunk to prevent ending pop
- Trim chunk tails before emission to prevent echo artifacts
- Update README with audio quality fixes documentation
Nighthawk