SelectTTS: Synthesizing Anyone’s Voice via Discrete Unit-Based Frame Selection
Ismail Rasim Ulgen*, Shreeram Suresh Chandra*, Junchen Lu, Berrak Sisman
*Equal contribution
Speech and Machine Learning Lab - The University of Texas at Dallas
Proposed Method
Proposed SelectTTS framework with the frame-selection method. In the frame selection, frames z1,z2,z3,z4 are chosen through subsequence matching and frames z7, z9,z6 and z10 are chosen via inverse k-means sampling
Comparison with Baselines
| Ground Truth | X-TTS | VALL-E | YourTTS | SelectTTS (no sub-sequence) | SelectTTS (with sub-sequence) |
|---|---|---|---|---|---|
How much reference speech do we need?
| Ground Truth | 30 seconds | 1 minute | 3 minutes | 5 Minutes |
|---|---|---|---|---|
Effect of different frame selection strategies
| Ground Truth | SelectTTS (only inv k-means (rand)) | SelectTTS (only inv k-means (avg)) | SelectTTS (inv k-means (rand) + sub-match) | SelectTTS (inv k-means (avg) + sub-match) |
|---|---|---|---|---|
Effect of vocoder fine-tuning
| Ground Truth | Vocoder (no prematched fine-tuning) | Vocoder (prematched fine-tuning) |
|---|---|---|
Inspiration credit
Baselines
- [1] Casanova, E., Davis, K., Gölge, E., Göknar, G., Gulea, I., Hart, L., ... & Weber, J. (2024). XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model. arXiv preprint arXiv:2406.04904.
- [2] Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., ... & Wei, F. (2023). Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111.
- [3] Casanova, E., Weber, J., Shulby, C. D., Junior, A. C., Gölge, E., & Ponti, M. A. (2022, June). Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In International Conference on Machine Learning (pp. 2709-2720). PMLR.