Software Quality

Welcome to the latest issue of Engineering Enablement, a weekly newsletter sharing research and perspectives on developer productivity. Subscribe to get new issues.

This week I read a new paper from Google‘s Engineering Productivity Research team sharing how they define Software Quality. For engineering leaders focused on quality, this paper provides a helpful framework for articulating what it means and where improvements need to be made.

Google’s team has previously talked about how they measure developer productivity through three dimensions: speed, ease, and quality. These counterbalancing dimensions important because they allows leaders to make sure that improvements in one area don’t come at the expense of another—for example, if they make a change that attempts to impact speed, they’re also going to be keeping a pulse on how it impacts quality and ease.

Of these three dimensions, quality is the hardest to measure because it’s also the hardest to define. Additionally, it means different things to different people, so conversations about “how do we increase quality” can lead to misalignment. In this paper, Google’s team breaks down how they arrived at a shared definition, the framework they landed on, and the specific metrics they track to measure it in practice.

My summary of the paper

To better understand what “quality” means, the research team started by conducting an extensive literature review to understand the themes that regularly appeared in papers on code quality. (They found several items, including defect rate, reliability, maintainability, and more.) Then, they conducted two series of interviews with developers at Google asking how they would articulate the different types of quality, their impact, and their relationships to each other.

Based on their interviews and literature review, the authors developed a “theory of quality.” It posits that there are four types of quality that influence each other: process quality, code quality, system quality, and product quality.

The four types of software quality

1. Process quality

Process quality refers to things like having comprehensive and deterministic testing, thorough code reviews, organizational consistency, and an effective planning process. In other words, factors of developer experience. The researchers say “everything begins with a high-quality development process,” because higher process quality leads to higher code quality. In fact, they point to multiple studies that have shown that process-based metrics are even more predictive of post-release defects than existing code quality metrics.

“There is good evidence that these measures can predict the overall software quality; multiple studies have shown that process-based metrics are more predictive of post-release defects than existing code quality metrics.”

2. Code quality

This type of quality is what most developers think of when discussing quality: it is the ease of working with and understanding code so that developers can easily make changes to it. In their interviews, the researchers found that developers most commonly related code quality to “maintainability” — this is consistent with the findings from another similar study.

The impact of code quality is twofold: it improves the quality of the system by reducing defects and increasing reliability, and it also leads to higher velocity for developers.

“In prior research, we found that developers’ perceptions of code quality were early indicators of their later perceptions of developer velocity. This is interesting because it highlights that our three components of productivity (speed, ease, and quality) are not always in strict tradeoffs with each other; in some cases, they can also amplify each other.”

3. System quality

System quality relates to reliability, performance, and defect rates. Measuring system quality is challenging because of the sparsity of data: outages and security incidents should be rare, so it can be hard to tell whether efforts to improve these areas are having an impact. For this reason, the authors suggest using process and code quality metrics since they are indicators of system quality.

“Without these intermediary metrics, stakeholders may think that a year with no outages is evidence that the system is high quality because they lack visibility into how developers are experiencing the code base and the ways in which it may be slowing them down.”

4. Product quality

Product quality is the end goal. It’s what is experienced by customers, and specifically refers to the product’s utility, usability, and reliability. This type of quality, in other words, is how well the product does what it says it will do. The authors note that engineers have influence over the aspect of reliability, however they need to work with product managers and user experience designers to contribute to usability and utility.

Final thoughts

Google’s theory of software quality, including both the four types of quality and how they influence each other, can provide leaders with a way to have more clear discussions about the topic. It also provides a framework for thinking through how improvements might be made and measured. For example, the authors say that improvements to process quality may be a good path to improve code quality. They also said that system quality might be better measured using process and code quality metrics. Ultimately, using this framework may help leaders and teams make sure they’re all referring to the same thing in conversations about quality.

Who’s hiring right now

This week’s featured job openings. See more open roles here.

• Realtor is hiring a Head of Developer Productivity | Austin, TX
• Amazon is hiring a Senior Programmer Writer - ASBX | Seattle and NYC
• Lyft is hiring an Engineering Manager - DevEx | Toronto, Canada
• The Hartford is hiring a Sr Staff Platform Engineer | Multiple cities, US

Thanks for reading. If you enjoyed this issue, consider reposting it.

-Abi