An Interactive System for Dual Narratives
- Aug 8, 2025
- 3 min read
Updated: 3 days ago
OVERVIEW
The project explores an interactive storytelling system using gesture tracking technology to let audiences phase between two perspectives in the same story. The logic allows them to blend between two perspectives using hand tracking, while switching between different chapters using the "pinch" gesture.
THE CHALLENGE
The project presented a self-imposed challenge to develop a system that facilitates non-linear storytelling using gesture tracking technology, continuing my research and experimentation with interactive storytelling techniques.
ROLE & FOCUS AREAS
I served as the sole designer, artist and developer for the experience, using creative technology, particularly gesture tracking technology, to facilitate accessible interactive storytelling for non-linear narratives.
SOLUTION & PROCESS
IDEATION
The intent was to create an interactive system that explores the dual point of view in the genre of non-linear storytelling. By engaging with the system, the audience can piece the story together through different perspectives and chapters.
Additionally, exploring all visible chapters unlocks a hidden chapter, giving the audience further clarity on the narrative.
The Leap Motion controller was chosen for its precision and low-latency tracking, making it suitable for a gesture-driven storytelling interface.
DESIGN
The project features a two-tier user flow system. At the top level, the audience can navigate between chapters, with the system randomising the sequence to misguide audiences, promoting multiple interpretations of the story.
Within chapters, the audience can blend between two perspectives to hear different accounts or opinions on a common subject.
Concerning the prototype’s interaction design, the audience can blend between perspectives via hand movement and switch between chapters using the “pinch” gesture, promoting ease of use.
DEVELOPMENT
TouchDesigner was chosen as the development software for its ease of use and flexibility with various hardware.
(1), (2) and (8) are node sequences that parse the raw data from the Ultraleap 2 sensor into signals to control navigation within the narrative.
(3) serves as a logical system to enable a hidden chapter.
(4) & (6) show audio and visual content for the chapters and perspectives. The visual contents were mapped to the respective chapters and integrated to blend between their perspectives.
(7) & (9) are node sequences that add a pixel dissolve effect to transition between the two perspectives in a chapter.
(5) is a node sequence to generate audio feedback when chapters are switched.
UNLOCKING HIDDEN CHAPTERS THROUGH LOGIC
The above setup randomises the chapter sequence (1) and validates when all visible chapters have been explored (2) & (3), following which, a hidden chapter is unlocked for the audience to explore.
STEREO AUDIO FOR BIAS
The use of stereo audio allowed the perspectives to be biased to the left and right outputs of an audio device, respectively.
EXPLORATION & REFINEMENTS
Through development, experiments and tests were done to refine the system’s interaction design and mechanics.
OPEN/CLOSED PALM TRACKING VS. PINCH GESTURE TRACKING
While tracking hand movement to blend between perspectives within a chapter remained constant, switching chapters used the “pinch” gesture interaction following issues with palm state tracking.
Adopting the “pinch” gesture approach reduced the number of times a chapter was switched unintentionally while blending between perspectives by at least 50%.
AUGMENTING THE RANGE OF HAND MOTION
Additionally, augmenting the range of hand motion to roughly 15 centimetres from the sensor’s centre point reduced the strain of blending between perspectives.
PROJECT OUTCOMES & NEXT STEPS
The project demonstrated my ability to craft a complete and intuitive gesture-based narrative system, continuing my personal explorations with interactive storytelling using immersive media.
Further experiments will focus on crafting a compelling storyline while improving the experience's spatial quality using a dedicated play space, larger displays, and better audio technology. Additionally, I am interested in adding more layers of logic to implement scenarios and conditions that facilitate non-linearity in the narrative experience.
