Q&A about testing software

In this article, I would like to summarise some of the questions that I've received from one of my colleagues at ThoughtWorks about testing.

I can totally see myself asking these questions and I'm sure that it might be true for other people too. Hopefully, reading this will help you solve some of your own doubts and advance you even further in your own understanding of testing software.

Of course, a little disclaimer is needed, the answers presented here are the result of my own experiences and a collection of my own thoughts, the objective of this article is also to improve my understanding of it, so please feel free to leave feedback if you find some of the answers to be imprecise or straight out incorrect.

• Should all the tests be put in the CI/CD pipeline?

Yes, the pipeline should run all the tests all the time. The CI/CD pipeline is your last resource before shipping the code, so it makes sense that it runs all the tests and necessary checks in order to ensure that the code, is working, is well-formatted and it’s safe (from security vulnerabilities and bugs). If in doing so the pipeline becomes slow and lengthy it means that some work needs to be done in improving that, either by using smart tricks like parallelization or by adding computing resources. Skipping the tests is not an option.

• Should unit tests always stub/mock the external services?

Yes, this is a pre-condition for a test to be actually unitary. To help the understanding of it you can try to call them, “Isolated” tests, in this way that they should be complete without dependencies, therefore isolated. A useful analogy that is frequently used is to imagine testing the engine of a car without the actual car. Doing so is also very convenient while developing using TDD when you can develop a piece of code without having to actually have the rest of the system even implemented.

• What should I do if a test function becomes too long? (that contains a lot of setup before the final assertion)

This depends, usually, the preference is for tests not to be very DRY. If most of the lengthiness is due to data setup, then it might make sense to extract it into reusable objects and modify simply the bits that affect you. If most of the setup is due to a lot of mocking instead that it is probably a smell that the design of the system under tests can be improved. Most of the time is because the code you are attempting to test is violating the Single Responsibility Principle, in more rare cases is because the design of the DB is not correctly normalized. If you find yourself in this situation a lot, it’s worth discussing it with the team in order to understand the root cause altogether.

• Is it important to reach a 100% coverage test in an application project?

Coverage is not a valuable metric, the sensible default is to have tests in place that give you the confidence to ship your code continuously and frequently, how this translates into a coverage percentage is not important. While working with TDD, you will find that you reach something close to 100% very often, but this is just a nice side effect, not something you should aim for. Joining a pre-existing project instead might be interesting to analyze the coverage percentage in order to understand which areas of the code are less covered, this might mean that something is hard to test and it’s very likely that most of the bugs of the system are coming out from that area of the code. Code coverage it’s a nice “conversation starter” or a good assumption to start investigating a bigger problem, never the final goal.

• Should we think about edge test cases? Or do we just need to finish the task to meet the ACs and leave these edge cases for another session?

You should always think about edge cases, but not be obsessed by them. Also sometimes make sense to tackle them immediately, sometimes to postpone it. There is no sensible default or silver bullet for this. Your understanding of how much and when to stop thinking about edge cases comes with experience, a healthy practice here is to confront a co-worker when in doubt, this will spark a nice conversation from which you will both learn a lot. All things said, quality delivery is a must, so we should very rarely trade quality in order to “complete” tasks, when we do so we should have a very good justification for it. If you find yourself with this internal debate a lot, it might mean that the KPIs that the team is measuring for defining success are wrong (ie: focusing a lot on velocity, will translate into a lot of low quality done tasks)

• Sometimes our e2e is testing the same scenarios with slight differences. Do we need to delete one of them?

Redundant tests have to be deleted and lean toward an approach that fails fast instead of over-testing. It’s way cheaper to fix a bug once in a while than having to maintain a lot of redundant testing code every day. All the code we write comes with a marginal maintenance cost and we need to take that into account. You should always look for the compound effect that redundancy adds to your development productivity. So if you have the feeling that some of the E2E are testing similar scenarios that it's worth having a conversation in more detail with the team. Also, a good rule of thumb for E2E tests is to focus on happy paths and not try to be extensive with them since they are the most exposed to flakiness and they are generally the most expensive (in terms of time) to run.

• Any sensible default/good practices/rules of thumb about TDD?

There are probably many, but for me, the absolute most important are the following:

1. Prefer outside-in TDD over inside-out
2. Keep the implementation very dumb at first, let the tests pass, then refactor later
3. Create tests data from the beginning and reuse that in every test, so that a change in the model is less disruptive

I will try to update this article from time to time as soon as new questions come to my mind, or if something pops up from the comments or somewhere else. Hopefully, this was useful and helped you reason about the same doubts I've had in the past.

NOTE: Special thanks to Ariess Lin for formulating these amazing questions and helping me in the writing of this article