Virtual Production Exhibit

with Unreal Engine nDisplay

I am currently working on a real-time, sensor-based interactive game exhibit project in Unreal Engine. Extending work I started at RIT, I am developing a virtual environment that integrates input from a custom wireless handheld device. The experience runs in real-time using Unreal Engine, and players control game objects by interacting with the handheld device. The device collects positioning data from radio frequency sensors and touch inputs from a built-in smartphone screen. The physical space includes five Makerfabs ESP32 UWB boards to create a location tracking network. Four of the boards are used as “anchors” and one as a “tag”. Anchor boards are placed at known coordinates within the room and are pinged multiple times a second by the tag board. Each ping provides the tag board with an array of distances representing how far away it is from each of the anchors. This array is then sent to Unreal Engine via Open Sound Control (OSC) and processes the data into a virtual 3D position, effectively mirroring the real world position of the handheld device. An nDisplay configuration displays the game across multiple projectors which are mapped onto a curved wall using VIOSO 7 to blend the displays together.