Speech Resynthesis from Discrete Disentangled Self-Supervised Representations

Abstract

We propose using self-supervised discrete representations for the task of\nspeech resynthesis. To generate disentangled representation, we separately\nextract low-bitrate representations for speech content, prosodic information,\nand speaker identity. This allows to synthesize speech in a controllable\nmanner. We analyze various state-of-the-art, self-supervised representation\nlearning methods and shed light on the advantages of each method while\nconsidering reconstruction quality and disentanglement properties.\nSpecifically, we evaluate the F0 reconstruction, speaker identification\nperformance (for both resynthesis and voice conversion), recordings'\nintelligibility, and overall quality using subjective human evaluation. Lastly,\nwe demonstrate how these representations can be used for an ultra-lightweight\nspeech codec. Using the obtained representations, we can get to a rate of 365\nbits per second while providing better speech quality than the baseline\nmethods. Audio samples can be found under the following link:\nspeechbot.github.io/resynthesis.\n