Automated Unity UI Testing using LLMs

Hey Unity developers! Regression Games is excited to announce an open beta for our new product, the Automated Unity UI Testing tool.

Get Started: https://docs.regression.gg/automated-testing/getting-started
Feedback Form: https://d7y6yysps34.typeform.com/to/o8U6FGtO

Have questions, comments, or issues? Join our Discord and let us know!

For games such as puzzle and mobile games, UI development can take up a large chunk of development time. These UIs are incredibly important to the success of your game — it’s the first thing players see, it influences how you make money (IAP UIs), and it allows players to customize their experience. It’s incredibly important to ensure that your UIs are bug free, performant, and operating effectively.

Playtesting is a good start, but it’s not perfect:

• Play-testers may not explore the entire state space of your user interface.
• Developers usually play-test their UI infrequently, rather than after every new change.
• During play testing, it may be difficult to catch issues like hidden errors, performance drops, memory leakage, network latency, and other non-visual bugs.

Main Features

With our Automated Unity UI Testing tool, you can have an automatic process which navigates through your UI using an LLM driven by a goal, with only minutes of setup and no coding required.

• No-code approach — just a few game objects to get automation up and running.
• A goal provided by you allows an LLM to navigate the UI with a specific intent and terminates once that goal is reached.
• Extracts UI information and screenshots for later review.
• CI/CD friendly, with platforms such as GameCI.
• Works with Unity 2D and Unity 3D (as long as it uses typical UI components).

To get started follow these steps:

1. Add our SDK from this GitHub repository to your Unity project (instructions included in README, and also coming to the Unity Asset Store soon).
2. Add the Automation Controller game objects to the scene you want to test.
3. Create a GPT bot following the instructions here.
4. Start the bot from the in-game overlay (which can be disabled in final player builds).

Once you integrate these pieces, the bot will navigate through your UI until it reaches your indicated goal while collecting screenshots and screen state information, which can be used for later analysis.

Coming Soon and Next Steps

We are releasing this product as a beta so developers like yourself can try it out and request features. Your feedback is crucial to us, but here are a few ideas we’ve been thinking about:

• Collection of logs, crash reports, memory performance, CPU performance, and other metrics in addition to screenshots and state information
• Extended support for UI interactions, such as drags, pinches, and non-button UI components.
• Further fine-tuning of our LLM approach to provide more reliable sequences of actions.

If you have issues setting this up, let us know in our Discord!

‍