Enriching the Virtual Experience: Designing an Interactive Experience Using Electrical Circuits

Abstract

In recent years, much progress has been made in the field of virtual reality: creating an engaging experience by fully immersing users in a virtual world. This project, however, explores a different approach: creating an engaging experience by bringing elements of the virtual world into reality.

This project explores different methods of bridging the gap between virtual and physical worlds using electrical circuit components. Methods of efficiently passing data between different programs using different programming languages are investigated. Electrical components, such as LED lights and sensors, are explored as potential tools to create unique systems of real-time user input and feedback.

This research is motivated by the Virtual Technology & Design sophomore design studio, VTD-253-254. In this course, students develop a creative process and explore different design mediums through a year-long multi-player game design project. This research investigates how to expand this project’s scope into the physical world by introducing students to electronics, equipping them with more tools to create unique and innovative designs in the future.

Project Design

The first goal of this project was to establish real-time communication between a virtual world and a microcontroller circuit. The virtual world is built in Unreal Engine 5.3.2, a widely accessible engine that is used in various industries. An Arduino Uno R4 Minima was selected as the microcontroller, as they are simple, inexpensive, and beginner-friendly.

The main challenge of this research project was establishing a communication pipeline between Unreal Engine and the Arduino board. Due to differences and limitations in software, establishing this connection required a multi-step pipeline using a Python script and the Firmata serial communication protocol.

Once a method to connect the circuit board was discovered, the first objective was to build a demonstration project. In this demonstration, a LED light lights up based on the user’s proximity to an object in the virtual world. This was later upgraded to 3 lights, with different colors indicating how close the user was to the target object. I then replaced the 3 lights with a single RGB LED, which changes its color as the user approaches the target. Each of these circuits was assembled using simple components on a breadboard.

Allowing a virtual world to interact with electronics in the physical world opens a new dimension of potential for user engagement and feedback, which can be used to create immersive virtual worlds, unique multimedia experiences, or interactive art pieces.

Connection Method

As of April 2024, there is no official solution that allows Unreal Engine to communicate with a microcontroller directly. The first step of this project was to design a communication pipeline that would be able to transfer data between them.

Third-party solutions were avoided for this project, as they can have unreliable support and can easily fall out of date.

Arduino code is only “run” once, at which point it is compiled and stored on the board as machine code. There is no ongoing connection with the source code or the computer that runs it. If an ongoing connection is desired, a serial communication protocol such as Firmata may be used. This protocol allows messages to be passed to the board through a USB connection. I used Firmata for this project, as it is well-documented and easy to set up with Arduino.

In order for a program to send messages to the board, it must also use the Firmata protocol. Unreal Engine is based in C++. Currently, there is no official C++ implementation of Firmata.

As a workaround, a Python script with the PyFirmata module can be used to communicate with the board, while reading input from another source. This source is a .JSON file which can be output by Unreal Engine. A Blueprint on the user character saves a .JSON file containing the distance to the target location every 0.1 seconds. The Python script then reads this .JSON file on its own 0.1 second timer, determines which color should be shown, and updates the lights on the Arduino board.

Conclusions and Future Directions

A real-time link between a virtual world in Unreal Engine 5 and a physical microcontroller circuit has been demonstrated. This technology has a lot of potential to be used in interesting ways.

Setting up the link between programs using this method requires several workarounds and is ultimately very indirect. Ideally, this process should be streamlined to remove potential points of error and pass data more efficiently.

If this technology were to be used in a student project, it would have to be well documented and kept up to date with not only the most recent version of Unreal Engine, but the most recent versions of Python, Firmata, and Arduino. The amount of software that must be installed and used in the connection process make the implementation of this technology fairly difficult and inaccessible to students in the current state.

Future research will look into how the communication pipeline can be streamlined by implementing different languages, protocols, and microcontrollers.

Additionally, further tests will look into how sensors can be used on the microcontroller to send input back to the engine. This could be used as a new form of input to trigger events in the virtual world.

Acknowledgements

This project was made possible by faculty mentor Dr. Yumna Kurdi and the Virtual Technology and Design program.

This project was funded by an OUR Spring Semester award 2024.