Spatial Visualization Tool Using MR & ML

Pradyumna Panikker
Jul 12
4 min read

Experience Type: Interactive Learning Experience in MR

Role: XR Designer and Production Designer

Platform: PC-VR/Mobile

Game Engine: Unity

Additional Tools: YOLO, Blender 2.8, Adobe Photoshop, Figma

This document is a proposal for the development of a interactive learning experience that leverages developments in the field of MIxed Reality and Machine Learning technologies. The experience intends to give students the ability to transport themselves to a different time period by interacting with physical artifacts from that time.

IDEATION

"A successor to the Littlewoods VR Archive experience."

This project is envision as a successor to the Littlewoods VR Archive, once again looking at objects as portals into the past. Adding to the original concept, the project will leverage Mixed Reality and Real-Time Object Detection to allow users to learn about the history of an artifact by physically interacting with it.

Concerning its use case, the project is intended to be an interactive learning experience that targets undergraduate students from the field of Art History and Archaeology, giving them the opportunity to better understand the artifacts they are examining by simulating spatial context.

"A more accessible tool to enhance student learning."

The rationale for creating the experience stems from the intent leverage to XR and ML technology to enable interactive learning for students to better understand their subject by establishing spatial context and atmosphere around the artifact being examined. Developing the experience as an MR application reduces the anxiety that comes with being isolated completely in a VR simulation. With MR, users will still have visual access to the physical space with the virtual elements blending with it.

EXPERIENCE USER FLOW

The experience begins with a training module that helps users familiarize with basic XR interaction techniques and navigating an experience Mixed Reality, ensuring they are better prepared to full experience the simulation.
Following the successful completion of the training module the learning sequence begins in passthrough mode with the hand tracking functionality enabled.
The user is then prompted to walk towards the table and gaze at any one of the three artifacts placed on it.

When the user gazes at the artifact, a circular progress bar appears, taking about 3-4 seconds to complete.

An immersive and interactive virtual environment corresponding to the artifact in focus spawns around the user.

The user is then prompted to look around and interact with various UI elements via gesture input functionalities.

The user can then gaze at another physical artifact, materializing the virtual environment pertaining to that artifact.

Alternatively, one can revert to Passthrough Mode by interacting with the button tethered to their wrist.

PROPOSED DEVELOPMENT WORKFLOW

This interactive experience is intended to be a standalone mobile MR application that integrates real-time object detection functionality using ML models to interact with physical artifacts.

"Creating Custom Data Sets Using YOLO."

The images will cover different angles of the artifact with similar lighting conditions as the space in which the session will occur. Once the images have been photographed, the subject in the images has to be annotated/labelled to be identified by the ML tool.

"Training the Custom YOLO Model."

The annotated images will then be used to train a custom YOLO model such as YOLOv5, YOLOv8 or newer, typically done using a desktop GPU using Python or PyTorch.

"Exporting the Model and Integration with Unity."

The trained YOLO model will then be exported as an appropriate format such as ONNX into Unity where it will be integrated into the interactive experience.

Unity Sentis will integrate the trained YOLO model into the interactive application being developed in Unity, while the Camera Passthrough API can provide access to the headset's camera feed, sending the live passthrough video feed from the headset into the YOLO model for inference.

POTENTIAL CHALLENGES & BOTTLENECKS

"Occlusion of the physical environment, posing a risk to the user and the artifact's safety."

Since a virtual 3D environment will be overlayed on top of the video passthrough feed, there is the risk of users losing visual access to the physical world, potentially increasing the level of anxiety and risk of physical harm during the experience.

This could be mitigated by fading in the virtual environment only beyond a certain distance from the center of activity, i.e, the position of the user and the table on which the physical artifacts rest. Visiblity of at the very least, a portion of the physical space, will ease the user's anxiety by retaining their connection to the physical space.

"Integrating real-tme object detection functionality into an interactive experience for the Meta Quest 3."

Development of this interactive functionality will require expert knowledge of ML tools. While there resources and templates have been made available, developing the experience will require sufficient support and guidance.

As an alternative, the programming can be simplified by using image recognition technology to identify QR codes corresponding to each artifact, thus triggering the respective environments.

"Optimizing detail virtual environments for standalone MR headsets."

Lastly, since this experience is intended to be a mobile/standalone interactive Mixed Reality experience, the 3D scenes will have to be extensively optimized to run smoothly during runtime. This would mean being frugal with dynamic components in the scene while keeping the experience presentable and clean.

NEXT STEPS

Currently, I am familiarizing myself with MR development workflows in Unity as well as use of the Camera Passthrough API by Meta. The first of this project would be to realize an alternate workflow that uses the Camera Passthrough API for image recognition, detecting specific QR codes corresponding to a specific physical artifact.

Following this, work will be done with YOLO to create custom data sets that can be then used to train an ML model to detect the target artifacts.

RESOURCES