Scaffolding for Object-Based Learning Using Mixed Reality
- 7 days ago
- 3 min read
OVERVIEW
The experience serves as a proof-of-concept for a learning and development tool to aid object-based learning. The project's objective is to develop an interactive tool that provides spatial context to an object, giving the user more knowledge about its history, applications and so on.
The prototype in its current state demonstrates an immersive and interactive system that spawns specific meshes when the user gazes at a specific QR code.
Note: The experience is currently in development with future plans for further refinements.
THE CHALLENGE
As the technology was relatively new, the challenge was developing a scalable system to implement the QR code detection mechanism to spawn specific objects after detection. This included implementing a system that effectively detects and identifies specific QR codes while providing the user with sufficient control over the experience's events.
As a personal exploration, the project presented challenges to develop my existing skills as a designer and technologist.
ROLE & FOCUS AREAS
As a solo project, I independently designed and developed the prototype, with the exercise focusing on UX design and development for Mixed Reality systems.
The prototype serves as a learning and development tool for object-based learning, providing users further context about a physical artefact using interactive visuals and sound.
SOLUTION & PROCESS
IDEATION
Originally, the idea for the prototype stemmed from a concept I had developed in 2025 while exploring Machine Learning technology's integration with Extended Reality. The original concept proposed training ML models using custom data sets of specific objects to build a library that serves as a reference to identify specific objects. The original concept and its potential method of implementation can be studied here.
Due to a lack of expertise in ML technology, the prototype in its current state relies on the use of QR code detection technology to identify the object and trigger a response.
DESIGN
From a UX perspective, the objective was to develop an accessible interaction system which allows users to comfortably trigger and cull events with the QR code detection feature.
Currently, the system is designed to contain two tiers of validation. A quick scan of a QR code spawns a prefab to trigger meshes or events via a timed gaze interaction. The originally spawned prefab also serves as a buffer to integrate two different features, i.e, the QR code detection feature and the gaze interaction system, into a single interactive sequence.
DEVELOPMENT
The prototype's development began with the setup of a project that facilitates the use of the QR code detection system. To aid programming, Perplexity AI was used to generate C# scripts for interactivity.
The system to spawn specific meshes using QR code detection and gaze interaction was programmed as follows:
Following the project's configuration, a script was generated to trigger specific "buffer" prefabs that would be tracked with the QR code.
With the buffer prefabs created and assigned to the script, a separate script to control the spawning of respective meshes or events through gaze interaction was created and assigned to each of the “buffer” prefabs. The script exposes parameters that allow manipulation of the time it takes to spawn and delete a mesh through gaze input.
PROJECT OUTCOMES & NEXT STEPS
The prototype in its current state serves as a proof of concept for a much more complex and sophisticated setup that can aid the teaching and learning process using interactive XR technology.
Future versions will expand upon the current structure to give users the ability to teleport and interact with virtual assets through gaze and hand-tracking mechanics.