revert: restore codec levels 1-15 embedding summing in finetuning

Reverts a change we applied from open upstream PR #178 (commit f83f184),
which removed the sub-codebook embedding loop arguing the talker shouldn't
see levels 1-15 during training. Investigation of the inference code
(modeling_qwen3_tts.py:1894-1920) shows the sub-talker generates levels
1-15 before the talker's next step, and all 16 levels are summed into
the input. Omitting them from training creates a train/inference mismatch.
This commit is contained in:
Kedara Studios
2026-02-09 20:05:23 +01:00
parent 32c00eb088
commit 233a141057
+5
View File
@@ -92,6 +92,11 @@ def train():
input_embeddings = input_text_embedding + input_codec_embedding
for i in range(1, 16):
codec_i_embedding = model.talker.code_predictor.get_input_embeddings()[i - 1](codec_ids[:, :, i])
codec_i_embedding = codec_i_embedding * codec_mask.unsqueeze(-1)
input_embeddings = input_embeddings + codec_i_embedding
outputs = model.talker(
inputs_embeds=input_embeddings,
attention_mask=attention_mask,