Google has introduced new Gemini features, allowing users to engage in conversations about YouTube videos, images, and files using natural language. This new feature is available to users with devices like the Galaxy S24, Pixel 9, and possibly other models.
With Gemini Live, users can discuss YouTube videos, PDFs, and images. The feature is activated through the Gemini overlay, which can be accessed by pressing the power button or swiping from a bottom corner of the screen while viewing the content. Once activated, users can choose to talk about the video or ask questions related to it. Similar options are available for PDFs and images, with an additional step of selecting the file by pressing a plus symbol. A button labeled “Talk with Gemini Live” appears to facilitate the conversation.
Gemini Live is particularly useful for summarizing YouTube videos and allowing users to ask detailed questions. For example, in a video by MKBHD about Samsung’s “Project Moohan” XR headset, Gemini explains the YouTuber’s likes and areas needing improvement. Users can ask specific questions about hardware or software, but only if the YouTuber has covered those topics.
For PDF files, Gemini can summarize content, answer questions, and even create quizzes to test users’ knowledge. However, users should be cautious with large files, as processing can take several minutes.
Currently, Gemini Live does not support direct conversations about articles on websites. Instead, it offers a brief summary of the article content. Like other AI models, Gemini Live may sometimes provide inaccurate information. During testing, it incorrectly identified an image of an elephant, highlighting the potential for errors.
To manage data privacy, users can control the automatic transmission of display actions to Gemini. By long-pressing any Gemini prompt, users can toggle this feature on or off. When disabled, the “Live Speak” button appears only after manual content submission. To re-enable automatic transmission, users hold the “Ask About…” chip and select “Enable Automatic Transmission.”
These new Gemini features are part of Google’s Project Astra, introduced during the I/O 2024 event. More components of Project Astra, like screen sharing and live video streaming in the Gemini app, are expected in the coming months. Long-term, these functions aim to integrate with XR headsets and AR glasses, as announced by Google and Samsung. Other hardware partners like Sony, Lynx, and Xreal are also involved.
Overall, Gemini Live offers a new way to interact with digital content, making it easier for users to engage with videos, PDFs, and images in a conversational manner. As the technology develops, it promises to enhance user experiences across various platforms and devices.