The XR community is abuzz with discussions about passthrough camera access, especially as companies like Meta, Apple, and Pico have already laid out their strategies. But the question remains: what will Google do with Android XR? After talking directly with Google, I can confirm they’re planning a solution much like what we see on phones. Curious to know more? Stick around!
The Camera Access Conundrum
Let’s roll things back a bit for those not yet in the loop. Standalone VR headsets these days double as MR headsets, offering an RGB passthrough view of the real world through front-mounted cameras. This innovative feature is what powers incredible mixed reality apps like Cubism and Starship Home, among others.
Developers, including myself, would love to get our hands on these camera frames. Imagine the possibilities of integrating AI and computer vision algorithms to interact with the user’s environment in groundbreaking ways. As I’ve previously mentioned, I’m a firm advocate for camera access as the key to unlocking true mixed reality. Only then can our applications become fully aware of the surroundings. For example, thanks to a workaround on Quest that granted me camera access, I managed to build a prototype AI+MR app for interior design, which would have been unattainable otherwise.
But this dream comes with a caveat: privacy concerns. A developer with malicious intent could capture a user’s entire environment unnoticed, potentially scanning sensitive personal data like ID or bank cards. Moreover, the risk extends to capturing images of faces or bodies for inappropriate uses.
Walking this tightrope between privacy protection and mixed reality innovation is a delicate affair. Balancing these two clashing priorities is vital.
XR Companies’ Responses
Initially, camera access was offered without much fuss. Those who remember my early work with camera textures on the Vive Focus might recall some fascinating experiments—diminished reality, Aruco marker tracking, and sound responsiveness.
However, as mixed reality gained traction, caution became the norm. Companies like Meta, Pico, HTC, and Apple began restricting camera frame access, driven by privacy fears.
This ban became standard practice until the XR development community voiced its collective need for this feature, urging manufacturers to reconsider. Notable voices pushed for transparent, user-consented camera access to run object recognition or computer vision algorithms. This raised the question: why are XR devices treated so conservatively when camera access on phones is granted with a simple permission?
Pressure from the community prompted companies like Meta to promise a “Passthrough API” launch, raising the question: what about Google and Android XR?
Android XR: A Phone-Like Perspective
Given Android’s predominance in the smartphone market, it’s interesting to see how Google aligns Android XR with Android apps. Speculation was rife, so I reached out to a Google representative for clarity on camera access with Android XR. Here’s the scoop:
Much like any Android app, developers can access camera frames on Android XR with user consent. They can request access to the world-facing camera, akin to the rear camera on smartphones, or the front-facing selfie camera. Both streams are accessed using familiar Android Camera APIs, Camera2 and CameraX.
When apps require camera access, permission, similar to what’s needed on phones, must be granted. Notably, the selfie-camera stream features the user’s avatar, created via Avatar provider apps/services on the headset, based on tracking data from OpenXR APIs for head, hand, eye, and face movements.
Consequently, Android developers can employ familiar classes such as CameraX for camera streams on Android XR headsets. These classes enable developers to capture frames, run machine learning analysis, and save media, just as they do on phones. This alignment with standard Android behavior heralds exciting possibilities!
However, while the avatar stream is available, access to raw camera streams remains restricted, at least for now. Hopefully, this will change in the future, especially for enterprise users.
For Unity developers, the narrative remains optimistic. Leveraging WebcamTexture for capturing camera frames might be possible, and if there’s trouble, experts could create a Unity-to-native wrapper for CameraX, granting access to the wider community.
Considerations for Android XR’s Development Stage
It’s crucial to remember that Android XR is still in preview, with no official headset release yet. Thus, changes could occur before the final launch. While unlikely, this possibility warrants consideration.
The Implications of Opening Camera Access
With Google and Meta lifting some restrictions, it’s likely other companies will follow suit. This could make 2025 a landmark year for mixed reality development. I’m eagerly anticipating the innovative creations developers will bring to life!
(Header image courtesy of Samsung)
Disclosure: This blog includes advertisements and affiliate links. Clicking an affiliate link earns me a small commission at no extra cost to you. See my full disclosure for more details.
Spread the Word
Related Articles
