A Matter of Perspective: Exploring a Dual Point of View in Interactive Storytelling

Experience Type: Immersive Storytelling using Hand and Gesture Tracking

Role: Author, XR Designer, Production Artist & Developer

Platform: PC

Game Engine: TouchDesigner 2022

Additional Tools: Ultrealeap Leap Motion Controller 2, Blender 3.6, ElevenLabs, Audacity

This article documents my iterative process of developing an interactive prototype that uses hand and gesture tracking technology to tell a story from a dual point of view. As with all my explorations with interactive storyteliing, my goal has always been to design and develop immersive and engaging stories that give the participants agency that is not generally feasible with traditional media.

The prototype allows participant to freely transition between contrasting perspectives concerning a subject, adding layers of meaning to the story.

IDEATION

"Giving the user agency to influence the narrative experience."

The prototype's development was a convergence of my learning and understanding of hand and gesture tracking technology and my continued exploration of interactive storytelling techniques. As with all my explorations with interactive narratives, the objective was to craft an experience which allows users agency to affect the narrative experience instead of being mere witnesses.

"Exploring interaction techniques that portray multiple perspectives."

The interaction mechanic's design, which uses the hand's positional data to swap perspectives on a specific subject was significantly influenced by the interactive storytelling methods featured in the award-winning game, Immortality by Half Mermaid Productions.

The primary interaction mechanics allow the player to hover over any character in a specific clip. Once clicked on, another video clip featuring the respective character in a different scenario or timeline is played, giving the player a different perspective or more information about a specific subject in the narrative.

Additionally, rewinding or fast-forwarding the video at a given point in time during the playback of the video gives a hidden perspective to the scenario or story featured.

The intent and the design of the Immortality's interaction mechanics were influential in conceptualizing and developing the interaction mechanic for the prototype with the intent to create a design that is simple and accessible.

THE NARRATIVE

While still in development, early explorations have compelled me to explore subjects like love and loss and time as the overarching themes for the narrative. This approach gives potential to create complex and contrasting perspectives that intend to confuse the audience and make them second-guess their perceptions of the story's characters.

PRODUCTION  

"Building a setup to transition between two points of view in a narrative."

The prototype, while primitive, is the first step in developing a interactive setup that explores how the audience can be given agency to piece a narrative together based on clues scattered across different chapters of the story.

The prototype was developed using TouchDesigner and the Ultraleap Leap Motion Controller 2 as the data-input device to parse hand-tracking and gesture information to manipulate the perspective.

"Refining the prototype's interaction design for efficiency and ease of use."

One of the objectives of developing the prototype was to refine the design of interactions that fade between different two perspectives and cycle through different chapters of the narrative.

In the first iteration of the design, the user used waved either hand horizontally to fade between perspectives and clenched their fist to cycle between chapters. Preliminary tests with users showed this approach to be inefficient, with users accidentally switch chapters while transitioning between perspectives. The video below demonstrates the same.

To improve the user's experience, the "clenched fist" gesture to cycle through the chapters was replaced with the "pinch" gesture, making interaction easier and more precise. The video below demonstrates the improvements in user experience.

NEXT STEPS

WIth a basic setup ready, the objectives moving forward include the development of a compelling narrative, both thematically and structurally that compliments the use of this interactive setup to create a compelling and engaging storytelling experience.

Storywise, the project's goal is to create an experience that allows the audience to develop the narrative incrementally, building their version by piecing together clues scattered across various chapters.

REFERENCES

• Barlow, S., Half Mermaid Productions. (2022) Immortality [video game]. Half Mermaid Productions.