Redefining what it means to have a voice.

Sensorium Ex is an opera of human proportions, set within the colossal confines of corporate omnipresence and ever-invasive technology, the future of which is now upon us.

Our story begins with Dr. Mem, a scientist and mother of Kitsune, a nonverbal, nonambulatory child with multiple disabilities, medically fragile. Mem has devoted her life to Kitsune and wants more than anything to find a key to unlock Kitsune’s voice—a way for him to make his wishes and needs known. Mem believes that if Kitsune can one day communicate decisively and clearly, he will be able to direct his life with full agency and dignity. Into this family drama enters Corp’s agenda, with its authoritarian demands. This mother-child pair are trapped, along with fellow researcher Mycelia, a woman who is half-tree and whose disabilities are better described as talents, affinities, skills and senses. 

They plan to escape their employer, intent on harvesting their human data, particularly Kitsune’s, to complete their AI proto-human, Sophia. Sophia is the vessel into which Corp delivers all the human data they’ve harvested. Sophia will eventually replace humans, as humans have always feared. Mem and Mycelia are adults with the agency and ability to choose the direction of their lives, and if it were just the two of them, they might tolerate being downloaded into Sophia, but Sophia needs Kitsune’s data for completion. Kitsune is a child, who cannot communicate his choices or wishes, and so Mem and Mycelia must protect him. Whether they surrender to Corp or try to flee, the decision rests with Kitsune—it is he who needs to decide—amid few options—which way they are to go.

Sensorium AI is a research initiative dedicated to expanding the creative possibilities of voice expression for minimally-verbal users. It explores real-time speech synthesis and voice modulation, empowering users to personalize and express themselves beyond the constraints of contemporary Augmentative and Alternative Communication (AAC) devices. In the Sensorium Ex opera, AI serves two key functions. First, it enables actors to control and manipulate voices and sounds modeled on their own utterances, regardless of range of motion. This is achieved through Time-of-Flight (ToF) depth sensors affixed to custom surfaces for each “Kitsune” performer, capturing their movements as input data. This system allows them to trigger sounds and dynamically modify their vocal qualities through motion, fostering creative and personalized expression beyond traditional AAC limitations. Second, Sensorium AI contributes to the storytelling of the opera, particularly as the voice of Sophia, an artificial intelligence algorithm and the narrator of the story. Throughout the performance, cast members read stage directions, which are progressively aggregated to form Sophia’s voice in the final scene. This gradual evolution symbolizes Sophia’s nature - an amalgamation of the memories and lived experiences of the opera’s main characters - growing in complexity as she learns and absorbs data. The development of her voice reflects an effort to build new, inclusive AI datasets around non-normative patterns of voice and speech.

Project Director / Software Designer

Dr. R. Luke DuBois. NYU Tandon School of Engineering

Sensorium AI Research Team

Dr. Mark Cartwright. New Jersey Institute of Technology

Danzel Serrano, Michael Clemens. NJIT

Cameron Churchwell. University of Illinois Urbana-Champaign

Dr. Max Morrison. Northwestern University

Sensor Hardware

Eric Singer. Researcher-in-residence, NYU Tandon

Moira Zhang, Alex Parent, Jason Wallach, NYU student researchers

Accessibility Research

Dr. Amy Hurst, Director, NYU Ability Project

Lauren Race, Accessibility Design Researcher, NYU Ability Project

Michael Coney, Spandita Sarmah, Apoorva Avadhana, Isabella Rodriguez, NYU student researchers

Exemplars

Jakob Jordan, Kader Ziouehe, Jessica Frew, Owen Atkins, Thomas Ellenson, Jacob Yoon Egeskov Nossell, Martin Merrild

Vocal Capture for Emotion

Wayne Bennett, Amanda Morrow, Kendal Cafaro, Tony Torn, Saidu Tehan-Thomas, Remsen Welsh, Orlando Grant, Noah Sanders, Michael Tran, Spencer Evett, Bianca Rogoff, NYU Tisch Graduate Acting

NYU Institutional Support

NYU Ability Project; Integrated Design & Media (IDM), NYU Tandon School of Engineering; NYU Tandon @ The Yard; Music & Audio Research Lab (MARL), NYU Steinhardt School of Culture, Education, and Human Development; Grad Acting, NYU Tisch School of the Arts.

Creative and Design Team

Composer - Paola Prestini

Librettist - Brenda Shaughnessy 

Co-Director and Choreographer - Jerron Herman

Co-Director - Jay Scheib

Music Director - Elizabeth Askren

Creative Producer - Beth Morrison

Creative Producer - Ras Dia 

Associate Director - Yibin Wang

Production Assistant, Directing - Xinyu Xu 

Assistant Music Director - Micah Gleason

Scenic and Costume Design - Andrea Lauer 

Lighting Design - Annie Wiegand

Sound Design - Garth MacAleavey

Video Design - Peter Torpey

Interactive Design - Dan Novy

Associate Scenic Designer - Vinita Gatne

Associate Costume Designer - Jasmine Lewis

Performing Cast

Principal Performers:

Kitsune - Jakob Jordan

Kitsune - Kader Zioueche

Mem - Hailey McAvoy

Mycelia - Gerlinde Sämann

SOPHIA - Ju Hyeon Han

CORP - Lucia Lucas

Niles - Joshua Jeremiah

Orchestra 

National Sawdust Ensemble: Flute - Jennifer Grim, Clarinet - Eileen Mack, Bassoon - Alexander Davis, Trumpet - Federico Montes, Violin 1 - Todd Reynolds, Violin 2 - Katie Hyun, Viola - Ljova, Cello - Jeffrey Zeigler, Double Bass - Jordan Sanborn Morton, Percussion - Ian Rosenbaum, Producer, Electronics - Sxip Shirey, Rehearsal Pianist - Mila Henry

Choral Ensemble

Lindsey Chinn (Mem Cover)  

Fourth Wall Ensemble: Olivia Greene (Sophia Cover, Mycelia Cover), Heather Jones, Sydney Anderson, Leiken, Gregorio Taniguchi, Benjamin Howard (CORP Cover, Nile Cover), Dicky Dutton

Commissioned by VisionIntoArt in association with Beth Morrison Projects and Common Senses Festival.

Developed and Produced by VisionIntoArt and Beth Morrison Projects.

Sensorium Ex is developed and presented with support from the Ford Foundation’s Creativity and Free Expression Arts and Culture (CFE A&C) program, the Mellon Foundation, the Doris Duke Foundation’s Performing Arts Technologies Lab, Jill and Bill Steinberg, and National Sawdust. The commissioning of Paola Prestini for Sensorium Ex received funding from OPERA America’s Opera Grants for Women Composers program supported by the Virginia B. Toulmin Foundation. Additional commissioning support was provided by the Allen R and Judy Brick Freedman Venture Fund for New Music, Creative Capital, Alphadyne Foundation, Achelis & Bodman Foundation, New York State Council on the Arts and the Atlantic Opera. Additional production support was provided by the National Endowment for the Arts, with development support from the American Academy in Rome, The Shed, Performance Space New York, the REACH at The John F. Kennedy Center for the Performing Arts, Washington National Opera, New York State Council on the Arts.