In the final weeks of teamLab Borderless at London's Saatchi Gallery, visitors walked through rooms where digital waterfalls literally parted around their bodies. The installation's underlying computer vision system tracked each person in real time, redirecting particle streams to flow around — not through — anyone who stood still. The experience felt uncanny: as if the building had become aware of you.
Borderless, the permanent version of which opened in Tokyo's Odaiba district in 2018 and drew over two million visitors in its first year, used a network of projectors, depth sensors, and machine learning models to create 50 interconnected artworks that responded to crowd density, movement speed, and even the collective mood of a room.
The word immersive has been diluted by marketing, but in performance studies it carries a precise meaning: an environment that surrounds and implicates the audience member rather than positioning them as a distant observer. Punchdrunk's Sleep No More (New York, 2011–present) is perhaps the canonical non-AI example — audiences roam a five-floor warehouse, encounter performers in any of 100 rooms, and construct their own narrative sequence. The architecture becomes dramaturgy.
What AI adds to this framework is responsiveness at scale. A human performer can react to one audience member at a time. An AI-driven environment can simultaneously adapt its visual output, soundscape, and even atmospheric qualities (some installations use scent diffusers on programmable schedules) to dozens or hundreds of visitors — each receiving a subtly individualized experience without any central scripting of those variations.
Traditional immersive theatre (Punchdrunk, Secret Cinema) uses space and simultaneity. AI-immersive adds machine perception — the space literally watches and learns from the people inside it.
Three technology families underpin most AI-driven immersive environments in 2024:
teamLab, the Tokyo-based art collective, is probably the world's most visited producer of AI-immersive work. Their Tokyo teamLab Planets installation in Toyosu (opened 2018, extended repeatedly, current run through at least 2027) processes visitor data to continuously adjust the density and behavior of projected koi fish in their famous wading pool. The fish actively avoid footsteps and cluster around stationary visitors — behavior produced not by hand-coded rules but by a trained simulation model.
In their 2022 teamLab: Reconnect installation in Tokyo, the collective incorporated biometric feedback for the first time: visitors wearing provided headbands had EEG readings fed into the visual system, causing their personal zones of the room to shift in color temperature based on measured brainwave states. This moved the technology from crowd-level sensing to individual physiological integration.
teamLab co-founder Toshiyuki Inoko has stated that the collective's goal is "art where the boundary between the artwork and the audience disappears." The AI layer is the technical mechanism that makes this philosophical claim architecturally real — not metaphorically but literally, as the visual content cannot exist independently of the bodies present.
AI-immersive environments generate significant data about audience behavior. Every footstep, pause, and gaze direction logged by depth sensors constitutes a behavioral dataset. Institutions like teamLab, Refik Anadol Studio, and the Barbican Centre (whose AI: More than Human exhibition in 2019 used audience-generated data to drive installations) have faced questions about whether visitors meaningfully consent to this data collection and how long behavioral logs are retained.
There is also a creative-authorship question: when the installation itself modifies its outputs based on what it has seen in previous sessions, is the artwork ever the same work twice? And who is its author — the programmers, the artists who specified its parameters, or the accumulated aggregate of every audience who shaped it through their presence?
You have been commissioned to design an AI-driven immersive installation for a 500-seat concert hall lobby. The space is used before performances, during intermission, and after — meaning crowd density and mood vary significantly across these windows.
In this lab, explore design decisions with the AI assistant: what sensors to use, how the space should behave differently at each time, what data is collected and how long it is retained, and how you would explain the system to visitors arriving for the first time.
In October 2023, the National Theatre's Immersive Storytelling Studio premiered Unreal City, a live performance in which audience members used smartphones to vote on narrative forks — but unlike simple branching-story formats, their aggregated choices were processed by a language model that synthesized new dialogue for the performers in near real time. Actors received earpiece prompts generated from the AI's synthesis of voting patterns, meaning no two performances shared the same precise spoken text.
The production's director, Noa Wertheim, noted that performers had to develop a new skill: accepting AI-generated lines without the preparation time normally afforded by rehearsal. Performers described the experience as "structured improvisation with a very fast outside dramaturg."
Before AI, participatory theatre already challenged the passive-audience model. Augusto Boal's Theatre of the Oppressed (developed 1960s–70s, Brazil) invited audience members — called "spect-actors" — to stop performances and suggest alternative actions for characters. Brazilian community theatre groups used this framework to rehearse social change scenarios with real participants.
By the 2000s, digital interactivity entered through voting systems (as used in You Me Bum Bum Train, London, and multiple site-specific UK works) and later location-based smartphone apps. But these systems branched between pre-written paths — the AI revolution is the move from selection to generation.
Pre-AI interactive theatre: audiences choose between options the writers pre-authored.
AI-interactive theatre: audiences generate inputs that the AI uses to synthesize new content — no author pre-wrote the specific output.
Several productions since 2020 have used audience smartphones not just for voting but for richer data collection. The UK studio Blast Theory, which has worked at the intersection of technology and performance since the mid-1990s, developed A Machine To See With (2010, updated versions through 2022) — a work delivered entirely through a phone call in which an AI voice character responds to the caller's spoken choices. The work predates large language models but used branching voice synthesis that felt, to many participants, indistinguishable from a human conversation.
In 2023, Blast Theory updated the piece with a GPT-4-based backend, replacing the scripted branching with genuinely generative responses while retaining the same framing: a voice calling itself "the machine" leads the participant through a heist scenario. The update radically expanded the range of participant inputs the work could accommodate — someone who had done the original scripted version reported that "for the first time, a weird question I asked actually got a weird, real answer instead of a graceful redirect."
German-Swiss theatre collective Rimini Protokoll has been documenting and theatricalizing systems — algorithmic, bureaucratic, biological — for over two decades. Their 2022 production Uncanny Valley featured a hyper-realistic android modeled on author Thomas Melle performing his own text about mental illness and doubles. The android's performance incorporated real-time facial animation driven by motion capture of an offstage performer, creating a hybrid in which audience members reported genuine uncertainty about whether the figure was human.
Critically, Rimini Protokoll did not attempt to conceal the technology — program notes, pre-show talks, and the android's own monologue all acknowledged its artificial nature. This transparency is itself a dramaturgical choice: the uncanniness was not a trick but a structured examination of where human performance ends and machine reproduction begins.
Rimini Protokoll's approach illustrates a key design principle for AI theatre: disclosed artificiality can produce more profound audience engagement than concealed artificiality. When audiences know the figure is a robot and still feel empathy — that emotional response becomes the artistic subject.
Productions like the National Theatre's Unreal City reveal that AI interactivity generates new demands on human performers. Acting training has traditionally emphasized preparation: learning text, rehearsing blocking, developing characterization over weeks. AI-interactive performance inverts this — the specific spoken content may be unknown until moments before delivery.
Theatre companies working in this space increasingly incorporate training in deep listening, status play, and responsive physicality — skills borrowed from improvisational theatre traditions (Keith Johnstone's work, the Annoyance Theatre's approaches) that prioritize present-moment attunement over memorized structure.
You are dramaturg on a 60-minute interactive theatre piece for an audience of 100. The piece concerns a near-future city deciding whether to replace its human mayor with an AI system. Audience members vote on policy decisions throughout; their votes are synthesized by an LLM that generates new dialogue for the performers — but the audience doesn't know in advance exactly how their votes will affect what they see.
Work with the assistant to design the participation mechanic: what do audiences vote on, how often, what signals to give them about consequence, and how do you handle it when the LLM generates something an actor cannot safely or meaningfully deliver?
At the 2022 Tribeca Festival, the immersive experience The Under Presents: Tempest made permanent its hybrid model: players using Oculus VR headsets shared a virtual space with human live performers — actors based in Los Angeles whose physical movements were captured and translated into avatar form in real time. AI moderation systems managed crowd flow within the virtual space and flagged behavioral anomalies, while human performers improvised responses to whatever avatar-audience configurations the system placed in front of them.
The work, developed by Tender Claws studio, was among the first commercially released VR experiences to successfully sustain live human performance within a virtual environment at scale, with multiple simultaneous performances nightly and persistent world states between sessions.
Mixed reality (MR) in performance describes work that blends physical and digital elements so that each influences the other. Unlike pure virtual reality (which replaces physical reality) or simple augmented reality (which overlays digital images on the physical world), mixed reality performance typically involves human performers whose physical actions have consequences in digital space — and vice versa.
AI enters this context in several distinct ways. First, as a rendering assistant: real-time AI upscaling and frame synthesis (techniques developed for gaming, now adapted for live performance) allow complex visual environments to run at higher frame rates on lower-powered hardware, expanding access to MR performance beyond major institutions. Second, as a behavioral layer: AI systems monitor where audience members are directing their attention (via eye-tracking in headsets) and adjust what information or narrative detail appears in their individual view.
In April 2021, the Royal Shakespeare Company premiered Dream — a live digital performance broadcast from an empty Stratford-upon-Avon stage into home audiences' screens, but with a significant interactive layer. Using Philharmonia Orchestra's AI Concert Hall technology adapted for theatrical use, audience members watching from home could influence the lighting and ambient soundscape of the performance through collective interaction with a shared digital interface.
Performer Oberon, played by Rufus Hound and later Katy Owen, inhabited a physical stage dressed with minimal scenery — the visual world was constructed almost entirely in real-time digital rendering, with the RSC's technology team using Unreal Engine to create an ever-shifting forest environment. AI systems managed transitions between states in response to performer movement tracked by a body-capture rig, meaning the forest literally grew and changed around the performer as they moved.
The production won a BAFTA for Interactive Creative Content in 2022 — the first live theatrical production to receive this award.
Dream demonstrated that mixed reality performance could preserve the liveness of theatrical performance — the unrepeatability, the risk, the presence of a live human body — while extending that performance into digital spaces accessible globally. The RSC subsequently established a permanent Immersive Experiences division building on this work.
The most radical potential of AI in MR performance is the capacity to give each audience member a genuinely different perceptual experience of the same event. In 2023, the artist collective Random International — known for the Rain Room installation (Barbican, 2012; MoMA, 2013) — premiered Framerate, a work in which AI tracked individual viewing patterns across a large LED installation and subtly modified the animation sequence visible from each viewer's position. No post-show conversation between viewers would produce a fully matching description of what they had seen.
This individualization raises a profound question for criticism and documentation. Theatre criticism has traditionally relied on shared experience — critic and audience member presumably watched the same event. When AI renders the work differently for each observer, criticism becomes partial in a new way: not the partiality of subjective interpretation but the partiality of literally different perceptual data.
Eye-tracking in current-generation headsets (Apple Vision Pro, Meta Quest Pro) can resolve gaze direction to approximately 1–2 degrees of arc — sufficient for AI systems to determine not just what a viewer is looking at but what within a scene is commanding their sustained attention, enabling fine-grained personalization of visual narrative emphasis.
Human performers working in MR environments face distinctive technical demands. Body capture rigs constrain movement; latency between physical action and digital rendering (even at 20–30 milliseconds) can disrupt timing-sensitive performance moments; and technical failures — a dropped tracking signal, a rendering crash — have no theatrical analogue. Productions in this space typically employ dedicated technical directors working alongside traditional stage managers, and develop specific rehearsal protocols for failure scenarios.
The RSC's Dream team reported that their most intensive rehearsal period was not of the performance itself but of failure recovery: what does the performer do if the forest disappears, if their avatar becomes desynchronized, if the audience interface drops? These protocols, once developed, become institutional knowledge that enables future productions.
You are directing a 10-minute scene within a mixed reality production. One live performer is on a physical stage. A global remote audience of up to 5,000 viewers accesses a digital rendering of that same space through web browsers — they see the performer's body-captured avatar moving through an AI-generated environment that responds to the performer's physical actions.
Design the key performance moment of this scene — what the performer does physically and what the AI-generated environment does in response. Then work through at least one failure scenario: what happens if the body-capture signal drops mid-scene?
The 2023 SAG-AFTRA strike, which lasted 118 days and involved 160,000 actors, was partly triggered by concerns about AI-generated digital replicas. The studios' initial proposal would have allowed them to scan actors' bodies and likenesses once — paying a single day's rate — and then use AI to generate unlimited performances of those digital replicas without further compensation or consent.
The settlement reached in November 2023 established new consent and compensation frameworks for AI replicas: actors must individually consent to each use of their likeness, digital replicas of background actors require compensation equivalent to working days generated, and studios must disclose when AI has been used in any final production. The live performance sector watched closely — because the same replica technologies are becoming available to theatre companies, opera houses, and immersive experience producers.
The economics of AI-immersive performance are deeply uneven. teamLab Planets in Tokyo charges ¥3,200 (approximately £17) per ticket — accessible by international standards but inaccessible to much of the world's population. The technology infrastructure for a production like the RSC's Dream requires multi-million-pound investment in engineering, motion capture, and real-time rendering — resources available only to major subsidized institutions or well-funded commercial producers.
At the same time, AI tools are reducing certain production costs. Generative AI soundscapes eliminate the need for live musicians in some contexts; AI-driven visuals can replace expensive physical set construction; and cloud-based rendering platforms (like AWS's Nimble Studio) make some real-time 3D production accessible to smaller companies at monthly subscription costs rather than capital investment.
AI-immersive performance can expand access — a work like Dream reaches global digital audiences who could never afford London theatre tickets. But it can also deepen inequity — when major institutions use AI to reduce technical staff costs while increasing ticket prices, the savings do not flow to workers or audiences.
When audience members enter an AI-immersive environment, they typically generate significant behavioral data: movement paths, dwell times, gaze directions, and in some cases biometric signals. The question of informed consent in these spaces is not straightforward.
Standard practice at most institutions is a brief notice in booking confirmation or venue signage — comparable to CCTV notices in public spaces. Critics argue this falls short of meaningful consent, particularly for biometric data. The UK Information Commissioner's Office (ICO) published guidance in 2023 specifying that biometric data collected in arts contexts is subject to the same GDPR Article 9 "special category" protections as biometric data collected commercially, requiring explicit opt-in consent and a clear purpose limitation.
Some artists have made data consent itself a performance theme. The Australian collective Chunky Move, in their AI-dance work Mortal Engine (updated performances 2019–2023), incorporated explicit consent rituals into pre-show protocol — asking audience members to actively indicate which sensors they permitted to observe them, with different consent levels unlocking different interactive features of the piece.
Chunky Move's tiered consent model — where more permissive consent unlocks richer interaction — offers a framework other institutions could adopt. It treats consent not as a legal formality but as a dramaturgical element, making audience members' relationship to surveillance itself an aesthetic experience.
The SAG-AFTRA settlement established consent and compensation principles for screen performance. The live performance sector has been slower to develop equivalent frameworks. In the UK, Equity (the performers' union) published AI guidelines in 2023 establishing that: members should not be required to participate in AI training data collection as a condition of engagement; digital replicas of living members require written consent for each use; and contracts must specify whether AI-generated content will supplement or replace human performance in any production.
However, enforcement remains challenging. Immersive experience producers — particularly smaller commercial companies — have signed Equity contracts while using AI-generated crowd atmosphere, ambiance, and background voices without flagging these as AI-generated content. The boundary between "production sound" (historically unregulated) and "performance" (covered by union agreements) is actively contested.
Real-time AI rendering for immersive performance is computationally intensive. A large-scale installation running continuous GPU-based generative visuals for twelve hours daily can consume energy equivalent to multiple households' monthly usage — often powered by data centers that may not use renewable energy. The carbon footprint of AI-immersive performance has received virtually no systematic study, despite the performing arts sector's broader sustainability commitments (the UK theatre sector adopted a Net Zero pledge in 2022 through Julie's Bicycle and the Tyndall Centre).
Some artists are explicitly addressing this gap. The artist Tega Brain's Solar Protocol project (ongoing from 2022) routes web-based interactive artwork through whichever server in a global network is currently receiving the most solar energy — making environmental constraint a design constraint. Applied to live performance, this approach would mean an AI-immersive installation's computational intensity visibly varying with grid carbon intensity — effectively making climate data part of the artwork's behavior.
You are the ethics lead for a new AI-immersive theatre company. Your company is planning its first major production: a 90-minute immersive experience using depth sensors, EEG headbands (optional), real-time AI rendering, and a generative soundscape. You expect 200 audience members per night across a 6-week run.
Your task is to draft key sections of an ethical framework document. Work with the assistant to address: data consent architecture, performer rights regarding AI replicas, environmental impact, and at least one equity consideration (who can and cannot access this work).