Is writing software tests worth it?

In this article we are going to talk about testing and software engineering and more specifically how important testing really is and whether or not you absolutely have to write tests when you’re writing software.

So how important is testing in software engineering? Here we are talking about tests that you write with code like unit tests or integration tests, we are not talking about manual testing where a human being, the developer or anybody else actually manually tests the software. So the really critical question that we want to answer here is do you have to write tests whenever you are writing software and the obvious answer is…
IT DEPENDS.

Some of you thought answer was going to be that you have to write tests all the time and maybe some colleagues of mine reading this article will be upset and thinking how could one of us say that writing tests isn’t absolutely mandatory. Well I’ll tell you why to understand why it’s not always mandatory to write tests in software to understand why “IT DEPENDS” is the obvious answer. We have to understand what the purpose of writing tests is.

So what is the purpose of writing tests? Well there are really two goals that writing tests has:

• the first goal is to presumably reduce the number of bugs that find their way into your codebase and to the feature or product that you’re shipping.
• the second goal is to presumably reduce the amount of time that human beings like the developers of the software or other people in your organization are going to spend manually testing the software. You can imagine that you’re gonna have to do some sort of manual testing if you don’t have written tests unless you want to ship your product blindly which would not be advisable.

So here you might be thinking isn’t the answer then obviously you should always write tests because clearly the two benefits of testing are really great, they’re very beneficial? Not quite because the keyword in those two benefits of testing is PRESUMABLY. You will presumably reduce the number of bugs in your application, you will presumably save people in your organization time from manual testing, but not necessarily.

Why not necessarily? Because writing tests isn’t easy. Writing good tests that are going to capture all the potential bugs in your software isn’t easy and is borderline impossible. As a software developer I had to write all kinds of tests: unit tests, integration tests, end to end tests, all kinds of tests for any feature that I developed and there were still bugs that found their way into the codebase. And then as far as reducing the time that you’re going to spend manual testing, that’s not necessarily gonna happen either.

There are some parts of your software that you’re never gonna feel fully comfortable not manually testing. Even if you have the most robust testing systems in place. As an example on many of our projects we have tests on a few of our services but even with those tests whenever we deploy big changes to production to our user interface or to some of our back-end services we always do some sanity manual testing especially for some of the critical flows like purchasing the product to make sure that the product or service is working correctly.

So the point is writing tests won’t necessarily save you that much time from manual testing because you might still continue manual testing, because you just don’t trust the machines and because quite frankly there are certain things that machine’s just can’t really catch but that humans can catch.

Another example here is on our project front-end applications we would always do manual testing on top of the rigorous tests that we had. We would do big bug hunts before launching features and we would always catch all kinds of issues that tests just couldn’t catch.

But okay I know what you’re thinking, writing tests still has some potential benefit even though it’s not necessarily gonna be the thing that saves your software from all bugs and make your engineers super productive, it can do that, therefore it must be worth it.

Once again not quite. Why? Because writing tests is not free, writing tests has a cost. What’s the cost of writing tests? TIME. Engineering time, engineering resources.

Like I said before writing good tests that actually catch a lot of bugs or prevent bugs from going to production is already not easy but on top of that writing good tests takes a lot of time. As an example for a given feature writing tests for that feature on top of developing it would quite literally double if not even triple the time that it took to build that feature out.

And this is for the simple reason that testing is difficult. Writing tests often involves using some kind of special framework, you have to mock services that can sometimes break, sometimes you have to debug tests and debugging them is a nightmare. The more powerful and useful your tests are the more difficult they become to write and to set up and to debug.

For instance if we’re talking about front-end tests there are a lot of different kinds of tests that you can write. Two common types of front-end tests are:

• Unit tests where you just test some function or maybe some component in your application like a button. Testing a simple button is likely not going to be too difficult. You can make sure that when you click the button some callback that you pass to the button component gets called. You can make sure that the button displays the correct text but maybe that test isn’t gonna be super useful.

• On the other hand there are also end-to-end tests. These are tests where you’re actually going to use some special software that opens up your browser and that is actually going to replicate what a user would do on your website. It’s going to navigate to pages, click on buttons, type in input fields. That’s going to be a much more powerful test, that might be a lot more useful and actually save you some of the manual testing later on but that’s going to be a lot more difficult to set up because it’s just a tougher test to write. You have to get the whole framework working, you have to make sure that it’s hooking up to the right back-end depending on what services you’re testing, you may not want to be using real production data. It’s just going to be a lot more complicated and it’s gonna take a lot more time and that is going to be costly.

The point that I’m trying to make is that writing tests takes time. Writing tests has a cost and when an activity has a cost doing that activity inherently comes with a trade-off. And when something comes with a trade-off that’s when you get to that “IT DEPENDS” answer that I gave earlier.

Is the trade-off of having your engineers spend more of their time to POTENTIALLY reduce the number of bugs that make it to production or POTENTIALLY reduce the amount of manual testing that they or other people will do worth it? Maybe. But IT DEPENDS. It depends on the type of software that you’re building, it depends on how fast you want to get that product in the market, it depends on a lot of things.

So what I would advise any engineer or manager or person running a company who’s debating whether or not writing tests for their software is worth it, but I would advise them to ask themselves is: Is that trade-off worth it?

So for instance if you are developing the kind of software where one bug finding its way to production is unacceptable, like the type of software that can put lives at risk, software for airplanes, software for medical devices then in those cases that trade-off is probably worth it and you should probably write tests. Similarly if you’re writing software for something that maybe doesn’t put lives at risk but is still critical like software that deals with users money or software that deals with legal related issues.

But if your software is just a social media app, the core user facing functionality, that’s not necessarily hyper critical. Now that’s not to say that your app shouldn’t have tests, I’m just saying that it might be an example where you might say maybe we value shipping some product really fast over writing the most robust tests. You can have parts of your services that have a lot of testing and others that have very little, if any.

Testing any part of your back-end that involves writing to a database you should test, so you actually use test databases and test those functionalities. On your front-end code you should test purchasing functionality, some really critical functionality but beyond that it hasn’t really made that much sense for you to invest a ton of time in testing.

Keep in mind that if you are small company you need to focus on delivering new features, respecting sprints and deadlines and oftenly you don’t have much time to spend covering all the functionalities, but at large companies where you’ve got hundreds if not thousands of engineers working on the same services or sibling services at once, new engineers coming – old engineers leaving, you need a lot more rigor in your testing to make sure that things just operate smoothly. That doesn’t mean if your are a small company you shouldn’t write tests, you should certainly think about them a lot but you should also be very intentional with the parts of code that you test because that trade-off of time for those potential benefits of writing them is a really important trade-off for you to think about and it’s not a no-brainer decision like it might be if you were writing software for let’s say a medical device that is a life-or-death situation.