Smartphone Data Visualized on Gemini Live Platform to a Certain Extent
Going Visual with Gemini Live:
Gemini Live, the chatty bud on Google's Gemini app, just got a serious upgrade. Now, it can provide real-time answers about what your phone's camera and screen are displaying! The update will first hit Google Pixel 9 and Samsung Galaxy S25 devices.
You've already been able to share photos and screenshots with Gemini, but the live, real-time feature is what sets it apart – it's like the AI sees the world directly through your eyes! This functionality was teased under the Project Astra name last year.
Samsung labels it as a "trusty pal always ready to lend a hand," while Google suggests you can use it for personalized shopping advice, troubleshooting broken goods, or organizing cluttered spaces. You can chat with Gemini Live about virtually anything in your camera's view.
The update is free on Pixel 9 and Galaxy S25 phones, with additional Android devices following suit soon. However, always-on access will require a Gemini Advanced subscription, though the specific devices that will receive the update remain unclear. No word yet on iPhone compatibility.
As always, be vigilant about the accuracy of Gemini's responses, and remember an active internet connection is necessary for this feature to work. To access Gemini Live, tap the button to the right of the input box in any Gemini chat (it resembles a sound wave). Following this, you'll discover two new icons at the bottom: one for the camera (the video camera icon) and one for your phone's screen (the arrow within a rectangle).
I've tested the new features on my Google Pixel 9 by asking questions I already knew the answers to, just to check for any unwanted hallucinations.
Gemini Live in Action:
First up, I tested the camera interface with a Severance episode on my laptop. Initially, Gemini Live mistook it for You due to confusing Penn Badgeleys with Adam Scott, but it quickly corrected itself. When I asked about a package labeled UN3481 (lithium-ion batteries), Gemini Live correctly identified the batteries but failed to provide additional information. Upon further questioning, it claimed the batteries were packed separately, which is incorrect – the correct code is UN3480.
Gemini Live was successful in giving me instructions on how to reset my Fitbit Charge 6 when I aimed my phone camera at it. This could be quite handy when troubleshooting gadgets, especially if you're unsure about model numbers.
Sharing your screen with Gemini Live is intriguing. The app transforms into a compact widget, allowing you to use your phone as usual while asking questions about the content on the screen. Gemini Live performed well in identifying apps and specific content like movie posters and band photos. It also accurately translated a foreign social media post for me.
Regarding a website displaying the recent Leicester v Newcastle soccer match, Gemini Live correctly stated the score, goal scorers, and team information present on the screen. However, when I asked about the match date, it miscalculated it as May 22, 2023, which was nearly two years in the future.
Gemini Live showed no issues with retrieving answers swiftly, but the quality of the results remains a concern. Despite the convenience of pointing the camera and asking "how do I fix this?," instead of crafting complex Google queries, errors persisted, requiring careful fact-checking.
In essence, this is an advanced version of visual search – you might have previously typed "UN3481 label" into Google for the same query. The traditional search results let you assess the trustworthiness and authority of the information. With Gemini Live, the closed-box interface makes thorough checking essential to avoid potential errors.
- The live, real-time feature of Gemini Live, found within the Gemini app on Google Pixel 9 and Samsung Galaxy S25 devices, provides a level of tech that allows the AI to instantly identify and answer about what the phone's camera and screen are displaying, making it an interesting entity for visual tech comparisons with other devices such as a Fitbit or a smartphone screen.
- Intriguingly, one can take screenshots of their Fitbit, point their phone camera at it, and use Gemini Live to receive helpful instructions, providing a potentially useful tech solution for troubleshooting fitness gadgets, especially for those unsure about model numbers.
- The authoritativeness of Gemini Live's responses can be questionable at times, as demonstrated during a test where the AI miscalculated the date of a soccer match by nearly two years, highlighting the importance of employing cautious fact-checking when using this feature.
