ENHANCING VIETNAMESE SPEECH SYNTHESIS IN AUDIOBOOK THROUGH XTTS-BASED TEXT TO SPEECH AND VOICE CLONING

Abstract

• This thesis investigates the enhancement of Vietnamese audiobook speech synthesis using the XTTS architecture, a multilingual zero-shot Text-to-Speech model capable of cloning unseen speakers from short reference audio. Although XTTS provides strong baseline performance, its default configuration is not fully optimized for the tonal and prosodic characteristics of Vietnamese, particularly in long-form narration. To address this limitation, the proposed approach introduces a Vietnamese-specific tokenizer, a refined text and audio preprocessing pipeline, and a selective fine-tuning strategy that updates only the GPT-based acoustic module while preserving the remaining pretrained components. Experiments are conducted on the Phoaudiobook dataset, a large-scale Vietnamese audiobook corpus, using 56,879 samples from 276 speakers for training, 998 samples from 34 speakers for validation, and 6,458 samples from 35 speakers for testing. Model performance is assessed using objective metrics, including Character Error Rate (CER), Speaker Encoder Cosine Similarity (SECS), and UTMOS for perceptual naturalness. The fine-tuned XTTSv2 model achieves a CER of 0.0709, indicating stable textual accuracy, a SECS of 0.9126, reflecting very high speaker similarity in zero-shot conditions, and a UTMOS score of 2.2141, demonstrating acceptable perceptual naturalness for audiobook-style speech. These results confirm that targeted adaptation of XTTS effectively improves Vietnamese speech synthesis while maintaining strong speaker identity preservation, providing a solid foundation for future research on Vietnamese dialects and expressive narration..