Reverts a change we applied from open upstream PR #178 (commit f83f184),
which removed the sub-codebook embedding loop arguing the talker shouldn't
see levels 1-15 during training. Investigation of the inference code
(modeling_qwen3_tts.py:1894-1920) shows the sub-talker generates levels
1-15 before the talker's next step, and all 16 levels are summed into
the input. Omitting them from training creates a train/inference mismatch.
The HF ForCausalLMLoss already shifts labels internally, but sft_12hz.py
was also manually shifting before passing to model.talker(), causing
double-shifting and progressively faster audio with more epochs.
Additionally, the sub-talker forward_finetune used ForCausalLMLoss on
already-aligned logits/labels, adding another unwanted shift. Replaced
with direct cross_entropy.
See: https://github.com/QwenLM/Qwen3-TTS/issues/179#issuecomment-3870059313
Streaming inference (voice clone) - repetition penalty:
Without repetition penalty, the model can fall into a degenerate state
where it keeps sampling the same codec tokens over and over. This
manifests as:
- Looping audio: the same syllable or sound fragment repeats endlessly
- Extremely long generation: instead of reaching EOS in ~200-500 frames,
it runs for thousands of frames (up to max_frames=10000)
- Apparent "slowness": a response that should take ~1s of audio takes
10-30s to generate
The fix works by tracking previously generated first-codebook token IDs
and penalizing them before sampling:
- Tokens with positive logits get divided by repetition_penalty (lowering
their probability)
- Tokens with negative logits get multiplied by it (pushing them further
down)
This nudges the model away from re-selecting the same tokens, so it
progresses through the text naturally and reaches EOS in a reasonable
number of steps rather than looping. Default is 1.0 (disabled) and is
exposed through the supported_params whitelist in
stream_generate_voice_clone() so it can be set via generate_config or
user kwargs.
Upstream sync (QwenLM/Qwen3-TTS):
- Bump version 0.0.4 -> 0.1.1 to match upstream release.
- finetuning/sft_12hz.py: weight sub-talker loss by 0.3 factor to
prevent the code predictor gradient from dominating the main talker
loss during SFT.
- finetuning/sft_12hz.py: remove sub-codebook embedding accumulation
loop (codec groups 1-15) from input embeddings, unnecessary and
harmful for finetuning convergence (upstream PR #178).
- finetuning/README.md: update recommended hyperparameters to
batch_size=32, lr=2e-6, num_epochs=10 for more stable training.
Kedara Studios