Microsoft’s Vall-E Could Find Use as a Way of Editing Speech

Microsoft's Vall-E program might be useful for manipulating audio.

Vall-E, a voice AI tool from Microsoft, is trained on 60,000 hours of speech—100 times more than current systems—from more than 7,000 speakers, the majority of whom are from LibriVox public domain audiobooks. It is also trained on "discrete codes derived from an off-the-shelf neural audio codec model." According to Ars Technica, Vall-E is based on a technology called EnCodec, which Meta unveiled in October 2022. It functions by listening to a person's voice, dissecting the data into its constituent parts, then applying its training to simulate how the voice would sound if it were saying various sentences. Vall-E can mimic a speaker's timbre and emotional tone even after only hearing a three-second clip. On GitHub, Vall-E voice recreation samples can be heard. Despite being based on such a little audio sample, many are actually fantastic, sounding almost exactly like the speaker. Although some of them seem a little more robotic and more like conventional text-to-voice software, overall, the AI is still rather good, and we can anticipate it to get better over time.

The research report, which is published at Cornell University, notes that "Experiment results reveal that Vall-E significantly surpasses the state-of-the-art zero-shot TTS system [AI that recreates voices it has never heard] in terms of speech naturalness and speaker likeness." We also discover that VALL-E is capable of preserving the emotional state of the speaker as well as the acoustic context of the acoustic cue in synthesis. By merging Vall-E with other generative AIs like GPT-3, Microsoft researchers believe it might be used as a text-to-voice tool, a method of editing speech, and an audio composition system.

Researchers claim to have trained VALL-E on 60,000 hours of English language speech from 7,000+ people on Meta's LibriLight audio library, which is hundreds of times more than existing systems. The voice of the target speaker must closely resemble the training data in order to be mimicked. This will enable the AI to read a specified text aloud while attempting to replicate the target speaker's voice using its 'training'. The media production sector, robotics, or customized text-to-speech apps can all employ the AI model. However, if used improperly, it could pose a threat. The business warned that the model may be misused to impersonate or spoof voice identification because VALL-E could synthesize speech while maintaining speaker identity.

Concerns have been raised concerning Vall-E's potential abuse, as there are with any AIs. Political impersonation is one example, especially when Deepfakes are used in conjunction with it. Or it might fool people into thinking they are communicating with friends, relatives, or authorities and divulging private information. Additionally, some security systems employ voice identification. Regarding the effect on employment, Vall-E would probably be less expensive than using voice actors. The hazards associated with Vall-E usage, according to the researchers, can be reduced. "To determine whether an audio clip was created by Vall-E, a detection model can be created. When advancing the models, we'll also take Microsoft AI Principles into consideration."