StageEye: Autonomous Stage Tracking R&D

Standard tracking systems frequently lose track of performer identity during live concerts whenever lighting flashes or dancers cross paths. StageEye investigated a dual-layer neural vision approach to resolve identity ambiguity.

A high-speed YOLO model detects all human bodies across the stage, while a dedicated Re-ID model extracts appearance embeddings to verify the lead artist via cosine similarity. The system preserves trajectory continuity even when dancers cross directly in front of the artist or when the artist turns away from the camera.

In real-world stage setups, moving fixtures are clamped onto truss pipes at arbitrary rotations, meaning a 50-degree tilt increase may swing downstage on one fixture but veer into the wings on another. Attempting full 3D inverse kinematics proved brittle in fast-paced production environments.

StageEye investigated direct 2D image-space calibration: an operator designates a 2D stage rectangle (ROI) over the camera feed, the system triggers an exploratory 50-degree tilt drop to establish a test light pool, and then executes discrete 5-degree steps across pan and tilt. The engine directly maps spot centroids to camera pixels while accounting for beam distortion geometry (circular at nadir vs elongated ellipses at acute angles).

Bench testing revealed that while mathematically stable, discrete step-by-step exploration was mechanically slow due to physical motor settling times (150ms to 300ms per step before capturing an unblurred centroid). As an open hypothesis for future research, a continuous rotational sweep at 60 FPS synchronized with DMX timestamps was proposed to dynamically interpolate boundaries without mechanical pauses.

StageEye communicates directly with professional lighting boards and media servers using native entertainment industry protocols, requiring no proprietary hardware translators.

The software broadcasts lighting positions over standard stage networks (sACN and Art-Net) at a deterministic 44Hz. For GrandMA consoles, direct network commands control dimmer intensity, fixture angles, and cues. Additionally, the system outputs transparent high-definition video of the performer over NDI, allowing media servers like TouchDesigner to project visual effects directly onto the moving artist.