The Single Best Indicator of Software Quality Is Automated Testing
In my hometown of Penang, near where I lived south of the Penang bridge, lay a Free Industrial Zone, home to factories like Intel, Broadcom, Kingston, Western Digital, Jabil, and many others. When I was fresh out of high school about to choose a degree to enroll in at university, I wanted to be an “Engineer”, as did many of my classmates. I, however, chose “Software Engineering” as it seemed like the most affordable route to become one, and the rest is history.
Fast forward to today and there is no hotter industry than software engineering, it seems. Unlike traditional engineering, however, certification isn’t required, standards are loose, and software varies wildly in quality.
Martin Fowler proposes this “pseudo-graph” of the tradeoff between “internal” software quality vs the ability to cumulatively deliver functionality as time passes:
The way I assess where lines cross is by canvassing the opinion of skilled developers that I know. And the answer surprises a lot of folks. Developers find poor quality code significantly slows them down within a few weeks.
He presents a similar pseudo graph in a separate article about Design Stamina:
Putting effort into to the design of your software improves the stamina of your project, allowing you to go faster for longer.
How do we know the quality of software?
The long answer is: It depends on the problem at hand, and evolves as new methods are introduced.
My short answer is: The presence (or absence) of automated tests in a codebase correlate strongly to software quality.
Here are some graphs describing the benefits of automated testing:
One from Arthur Hickens, quoting a book by Capers Jones, on the exponential cost of repairing defects the “farther to the right” in the testing cycle that the defect is spotted.
Another from SimForm, in a graph that should look very familiar indeed:
Why does the presence of testing correlate strongly with software quality?
1. The way software is written is different in the presence of automated testing.
- Reusable, shorter functions are valued over long, duplicitous ones because they are far easier to test.
- Dependency injection is promoted because it helps with mocking.
2. Automated tests act as “truthful” documentation on function use.
Forget verbose READMEs, docblocks, and *shudder* Confluence-as-documentation: passing test cases objectively prove and show how code should be used — an invaluable asset in a time of high turnover and the Great Resignation.
3. Tests teach developers to think about edge cases
Can or should a sum(a, b) function handle negative arguments? If using a dynamic language, does it reject strings?
4. Tests mean the codebase is reproducible
I’ve yet to see a team adopting software testing that doesn’t also introduce automated deployments or artifacts, which saves even more dev-hours.
Being able to run tests against a CI/CD pipeline also requires devs to know how to bundle, install, and run their app, which is even better for expanding the DevOps practice and the goal of “You build it, you run it”.
5. Codebases with tests WILL be able to refactor with confidence, codebases without tests won’t.
And this means that the codebase quality will never be improved, ever. For the software quality to improve, entire swathes of code will need to be written alongside the brittle, existing functions, or the codebase must be replaced entirely.
“Let’s not touch this function in case something breaks, even though it can clearly be improved” — every engineer in a codebase without tests, eventually
6. Organizations that invest time in automating testing are forward thinking
Software development is a marathon, not a sprint (well, a series of sprints). Committing the additional time needed for testing, away from hammering away at code, signals that an organization or team has learnt to look ahead, and invest in the future.
Not the only indicator, but the single most important one
When I was thinking over this topic, some other best practices did come to mind:
Infrastructure-as-code: Very important, but I’d argue this relates more to reliability and deployment agility, not the quality of the underlying software.
Code reviews/SOLID/Code best practices: All of these are pretty subjective. Code reviews in particular are commonly misused to be some kind of “approval” process, rather than something to promote quality objectively.
So why don’t more teams do it?
If you recall the graph above, a valid scenario is one where a project simply won’t be long-lived enough, or won’t need to be maintained by enough people for there to be a payoff for the initial time invested — I’d also stress that the initial time investment gets shorter and shorter the more you upskill your teams. And that automating end-to-end testing is a great benefit when it comes to migrating off legacy systems (and all systems become legacy systems).
The other, all-to-common scenarios are where the team, tech leaders, or organization as a whole are not experienced or incentivized enough to plan for the long term, which I hope this article tries to address.
