How we used testing in our Unity3D game
In the following lines I’ll try to explain what Unity tests are, how to build them and how we used them in our puzzle game. You can see a few of our PlayMode tests running in the following video.
Testing possibilities
In Unity3D we have two options to test our code: Edit Mode tests and Play Mode tests.
According to the documentation: Edit Mode tests (also known as Editor tests) are only run in the Unity Editor and have access to the Editor code in addition to the game code.
In other words we can test custom Editor code, that we wrote, but also classic C# code that doesn’t include Unity specific code. By Unity specific code I mean code related to GameObjects, physics, rendering etc.
Because they don’t need the Unity Player in order to run, Edit Mode tests are extremely fast (milliseconds) when compared with Play Mode tests (seconds). But more on that later.
This testing speed gain is one of the reasons to split the logic away from the Unity bits of code. So basically write classic C# to handle the game logic. By doing this, we get modular code that can be tested using EM tests. This saves us a lot of time when running tests.
We’ll get to examples of how to write Edit Mode tests and how we used them in our game, later in this article. First let's talk a bit about Play Mode tests.
According to the documentation: You can run Play Mode tests as a standalone in a Player or inside the Editor. Play Mode tests allow you to exercise your game code, as the tests run as coroutines if marked with the UnityTest attribute.
This means that the Play Mode tests run in the Unity Player (Desktop/Mobile). These tests will test the Unity specific code and we can see them in action on the computer or on the phone. As mentioned before, these tests take a significantly greater amount of time when compared to EM tests. They are useful to see how a certain feature works (like dropping a ball) or test a game level on autopilot (what we actually do in our game).
One way to look at EM vs PM tests could be to see the Edit Mode tests as unit tests and the Play Mode tests as integration tests, with similar benefits. This is not really accurate, but as I said it is one way to look at it.
Getting started with Edit Mode tests
To create Edit Mode tests you must place them in the Editor folder. In the picture bellow you can see our setup with an EditMode folder for the EM tests and a PlayMode folder for the PM tests. The only rule is that you have to have the tests in the Editor folder. The subfolders are just a way of better organising the tests.
The next step is to create an assembly definition. You can find out more about that here. As you can see in our asmdef, we also reference TextMeshPro and the main ScriptsAssembly because our PlayMode tests requires access to both in order to run.
The next step is to actually create some tests.
How we used EditMode tests in our game
To give the next few lines some context, our game has some square-ish buttons that you press to trigger a panel of colorful balls. You can see them in the screen shot bellow.
The number of balls in that panel varies between 1 to 6 balls. In order to compute the position of the panel and of each ball inside the panel, we wrote some non-Unity specific code. For this example we’ll focus on computing the ball positions inside the panel.
To compute the ball positions inside the panel we wrote a class called BallGUIProcessor. This class has a method:
public float[] GetBallGUIXPos(ScriptableObj.FancyColor[] balls, float scaleFactor)
This method has an input parameter, named balls, that is basically an array of structs that contain the colors of each ball. This means that if this array has 5 elements we have 5 balls. The second parameter called scaleFactor specifies the scale of the balls. This is necessary because some levels have a smaller scale than others and the balls need to be scaled accordingly.
This method returns an array of floats which represent the x position for each ball. So for an input of 5 balls it will return an array of 5 floats.
What we needed to test, was that for any number of balls in the permitted range (1–6), the resulting positions are correct. This is a classical example of an EditMode testable method as it doesn’t contain any Unity specific code.
Here are a few lines from this test:
public class GUIBallProcessorTest : IPrebuildSetup {
private static ScriptableObj.ColorPalette palette;
private static float padding = 0.2f;
public void Setup() {
palette = (ScriptableObj.ColorPalette)
AssetDatabase.LoadAssetAtPath(
"Assets/Scripts/Data/Colors/Palette.asset",
typeof(ScriptableObj.ColorPalette));
}
[Test]
public void TestThreeBallsPositions() {
var colors = new ScriptableObj.FancyColor[] { palette.Red,
palette.Yellow, palette.Green };
BallGUIProcessor processor = new BallGUIProcessor(padding);
var xPositions = processor.GetBallGUIXPos(colors, 1.0f);
Assert.That(xPositions[0], Is.EqualTo(-0.8f)
.Using(FloatEqualityComparer.Instance));
Assert.That(xPositions[1], Is.EqualTo(0.0f)
.Using(FloatEqualityComparer.Instance));
Assert.That(xPositions[2], Is.EqualTo(0.8f)
.Using(FloatEqualityComparer.Instance));
}
}
Because we need to manage some initial setup, we implement the IPrebuildSetup method. Now we can override the Setup() method and initialize what we need. In our case we initialized the palette object, which contains some custom color related code. For the test, the palette object is basically a regular enum.
The method TestThreeBallsPositions(), as its name implies, sees if the xvalues supplied by the processor are correct. The requirement for such a test method is to write the [Test] attribute before the test method. The fact that we started the method name with the word “Test” is just a personal preference, it is certainly not necessary.
In the test method’s body we supply three balls (red, yellow, green), initialize the processor with a predefined padding of 0.2f (the distance between the balls). We then call the GetBallGUIXPos method with the 3 ball colors and a scale of 1.0f. We then assert that the results are equal to the expected positions, in this situation -0.8f, 0f and 0.8f. These positions are relative to the container in which the balls will be placed. So it doesn’t matter where on the screen the ball panel will be placed, the positions for 3 balls with 0.2 padding and a scale of 1.0, will always be the 3 aforementioned positions.
Did EditMode tests help us?
The previous test helped us quite a bit, as the GetBallGUIXPos method implementation was re-written quite a few times and the tests made sure that we made no mistakes in the process. As it turns out we did make a few mistakes along the way and this test helped us see that in a few milliseconds (0.06 seconds as you can see in the image bellow).
If we didn’t have this test, the alternative would be to go through various levels to test if the ball panel looks alright. The fastest way to do that would be to play a level with 6 balls and use them to see how the panels look as the number of balls changes. This would have taken a lot more time to test.
This is just one small example of how EditMode tests can help us. Unfortunately we couldn’t test as much as we wanted with EM tests because of the nature of the game and the game’s architecture which didn’t make it easy to test parts of the game in this fashion. (lesson learned and we’ll pay more attention to this next time)
In order to test the rest of the bits, we used PlayMode tests.
Getting started with PlayMode tests
When writing a PlayMode test we need the UnityPlayer in order to run the Unity specific stuff. In this situation instead of having the [Test] attribute in front of a test method, we instead have the [UnityTest] attribute.
All PM tests are coroutines so they have IEnumerator as a return type. We can implement IMonoBehaviourTest and test our MonoBeaviours. You can see a small example in the Unity documentation here. In order to run this test the UnityPlayer is initialized and you can see what’s happening on screen. We can do this on the desktop or on our phone or whatever is our platform of choice.
How did we use PlayMode tests in our game
We created our PM tests in the PlayMode folder of course. We didn’t test individual component behaviour in our PM tests, instead we tested each level.
Our game, being a puzzle game, has very strict rules for winning and loosing. It also has many individual components that create effects that in the end change the landscape of the puzzles. Because things got complicated really fast, we decided to test each level for win conditions and lose conditions, as well as special edge cases that might make balls spin out of the level or get stuck somewhere.
Each level has a test class that inherits from LevelCompleteTest. This is the super class that contains various primitives like WaitForSeconds durations: SMALL_DURATION, REGULAR_DURATION, LONG_DURATION and others.
It also contains a LoadScene() method that loads the main scene and a LoadTestLevel(int levelIndex) method to load the level to be tested.
In addition, it also contains methods to press on each ball selector, methods to press on a certain ball, methods to press on the rotate left and rotate right buttons. This group of methods are the means to interact with the level under test.
Finally the LevelCompleteTest class also contains methods to assert the outcome of the level solving: AssertThatYouWon() and AssertThatYouLost().
We didn’t start testing the game by writing this super class. We started writing tests and after testing a few levels, we realized that we can extract and reuse bits of code.
In the following lines you can see the code for testing the win condition for level 7:
public class Level7_CompletionTest : LevelCompleteTest {
[UnityTest]
public IEnumerator TestLevelComplete() {
//Load Scene
if (LoadScene()) {
yield return SMALL_DURATION;
}
yield return LoadTestLevel(6);
//Press the ball selector indicator
yield return PressFirstSelector();
//Press the yellow ball
yield return PressBall(1);
//Rotate level to the left
yield return RotateLeft();
//Rotate level to the right
yield return RotateRight();
//Rotate level to the right
yield return RotateRight();
//Press the ball selector indicator
yield return PressSecondSelector();
//Press the red ball
yield return PressBall(0);
//Rotate level to the left
yield return RotateLeft();
//Check if you won
AssertThatYouWon();
}
}
As you can see, the test code is quite easy to read and understand, thanks to the LevelCompleteTest super class. A thing to notice is that in these PlayMode tests you have to do some waiting. One such case is after loading a scene where we have to wait for a SMALL_DURATION to give the scene time to load.
In the more complex level tests, especially the levels from Chapter 2, we need to do a lot more waiting. Because we have to wait for the balls to fall, for the levels to rotate multiple times and other operations, the tests for complex levels tend to be full of yield return LONG_DURATION.
For levels with long complex solutions, this can take more than the default timeout of 30 seconds. This happened a lot, even though we run our tests at 8 times the normal speed (Time.timeScale = 8.0f). Taking more than the default timeout made our tests fail. To fix this, we added a custom timeout of 50 seconds by using the attribute [Timeout(50000)] before the guilty test methods. We reached this value after testing a few levels and realized that it’s enough to make our tests finish.
Did PlayMode tests help us?
The PM tests helped us a lot. We often had to introduce new functionality that even though is isolated from the rest of the game did require some code changes in determining win conditions and other behaviour. This in turn caused issues with previous levels. By having tests for every level before the one with the new feature, we caught the issues immediately instead of wasting tons of time with manually testing the previous levels.
Sometimes a new feature worked perfectly for the first level that used it, but not so much for the next level that used it in a more complex way or just in a different way. This in turn required heavy rethinking of the feature. After making the feature work for this more complex level we had to make sure that it still works for the previous simpler level. In this case the tests came to the rescue again.
Without the PM tests I sincerely believe that we would still be working on finishing the game.
As mentioned before, the main downside of the PM tests is the long execution time. As you can see in the screenshot below, just the TestLevelComplete method for level 7 took nearly 4 seconds to run. All the tests for 40 levels amount to over 15 minutes, and that is quite a lot of time.
Conclusion
This is our second Unity game, and the first one we thoroughly tested. After this experience I can’t insist enough that tests are a vital part of development. Even though it seems like wasting precious development time (it took us on average 40 minutes per level to write just the PM tests), in the long run we saved hundred of hours.
Even with all these tests in place, we still found and fixed quite a lot of bugs post launch. We still find and fix bugs, but I can only imagine what a disaster this would’ve been without the tests.
The most important lesson that we learned is that our next game should be better thought out, from an architecture standpoint, so that we can have much better EditMode test coverage. This game is pretty weak on that front.
We honestly hope that this article made you consider writing tests for you own games.
If you want to checkout our game (first chapter is free), you can do so on the PlayStore and AppStore.