In collaboration with Todd Hennessey at the Centre for Research and Innovation in Corner Brook, Newfoundland, DCXIX is developing a mixed-reality entertainment experience like no other. Project Moonstone is a linear narrative, virtual reality show incorporating live performance and audience participation that achieves a unique, shared-entertainment event. A virtual host, animated from real-time motion capture data, guides the audience through a series of ghost stories as collected and told on the Corner Brook Ghost Walk. The ambitious project streams live performance capture data into the Unity game engine, where it is synchronized and broadcast to participants for a shared virtual tour of Corner Brook's haunted history.

The final production, Corner Brook Ghost Walk: Live in VR!, revolutionizes theatrical storytelling with a novel use of virtual reality. The audience partakes in a shared retelling of stories from a local venue while virtually experiencing a re-creation of this same location from a past time. The development team at DCXIX has worked from archival references to recreate mid-century Corner Brook in the Unity game engine, and delivers an experience that is both entertaining and culturally significant.

The production setting is a re-creation of Corner Brook’s West Street, with the audience experiencing the virtual scene through a VR headset. Members are seated as if to watch a performance in real life, but through the headset, are instead seated and travel along West Street on the Corner Brook road train.

To create an accurate depiction of West Street in the virtual world, the developers used geospatial and satellite data to map out the geographic features of downtown Corner Brook. Archival photos and accounts were referenced for building a low-poly rendition of Corner Brook circa 1950.

Streaming performance capture data in real-time to a virtual environment may seem like a common use-case for this technology, but the reality is that there are no existing turn-key solutions for artists and storytellers. DCXIX overcame a number of technical challenges associated with streaming this data across networks and retaining enough accuracy for the target quality. What results is a viewing experience enhanced by technology but retaining the presence and immediacy of a theatre performance.

Performance capture data is also used for the animation of various characters encountered in the world. These pre-recorded ‘vignettes’ undergo a standard motion capture production process, with clean-up, rigging and retargeting as required. The animations can then be triggered manually or automatically through a custom event system developed by DCXIX.

Moreover, the scene lighting, sounds, and the motion of the train are all handled through the event system, and can be customized and overridden as required based on the presentation. These trigger controls are part of a larger in-development solution to provide creators with their own customizable control panel for virtual stage management.

The live performance aspect to this virtual reality piece is captivating and impressive. Typically, existing commercial experiences where viewers can see others in virtual spaces tends to be clunky with preset movements and emotes. The infinite variety of performance capture data from a sophisticated capture system enhances these interactions, and provides full character expression for the performer, underscoring the notion that a real person exists on the other side of the virtual environment.

In this way, Project Moonstone pushes the boundaries of theatrical performance and virtual reality interactions in order to experiment with and deepen our ability to empathize in virtual spaces.