Nvidia has introduced a new AI technology called Fugatto for generating audio, which is claimed to be more versatile and superior to all competing services. This technology can transform existing audio recordings, such as turning a piano piece into a vocal performance. It can also modify a voice recording to change the accent or emotion of the speaker. The technology is intended for music production, game development, and for “ordinary people who want to create things,” explains Bryan Catanzaro from Nvidia.
Fugatto, which stands for Foundational Generative Audio Transformer Opus 1, was trained exclusively with material under open-source licenses, according to Nvidia. The technology is controlled using text commands (“prompts”) or with audio files. In a demonstration video, Nvidia shows how Fugatto generates the sound of a passing train from a simple prompt, which then transforms into an orchestral recording. In other examples, the technology separates a voice from a song and generates another voice that recites a given sentence. Additionally, instruments can be added to an uploaded music piece.
“We wanted to create a model that understands and produces sounds like humans do,” explains Rafael Valle from Nvidia about the product. Approximately a dozen people contributed to its development. According to the news agency Reuters, there is still internal debate about whether and how the technology will be made publicly available. Catanzaro explains the hesitation by stating that every generative technology carries some risks: “We need to be careful with it, and therefore we have no immediate plans to release it.”
The development of Fugatto represents a significant advancement in audio technology. The ability to transform and create audio content with such precision and versatility opens up new possibilities for various industries. For musicians, it offers a tool to experiment with sounds and compositions in ways that were previously unimaginable. Game developers can use it to create immersive soundscapes that enhance the gaming experience. For everyday users, it provides an opportunity to engage with audio creation, making it accessible to a broader audience.
Despite the excitement surrounding Fugatto, Nvidia’s cautious approach to its release is understandable. The potential for misuse of such powerful technology is a concern. Issues like copyright infringement, deepfake audio, and the ethical implications of altering voices and sounds need to be carefully considered. Nvidia’s decision to delay the public release indicates a responsible approach to addressing these challenges.
As the technology continues to evolve, it will be important for Nvidia and other companies in the field to establish guidelines and safeguards to ensure that such innovations are used ethically and responsibly. The future of audio technology is promising, and with careful management, it can lead to significant advancements in how we create and interact with sound.
In conclusion, Nvidia’s Fugatto represents a groundbreaking development in the realm of audio generation. While its full potential is yet to be realized, the possibilities it offers are vast and exciting. As with all technological advancements, it is crucial to balance innovation with responsibility, ensuring that the benefits are maximized while minimizing potential risks.