Threshold for Size and Complexity Metrics: A Case Study from the Perspective of Defect Density

Editor's Notes

• #2: Hi everyone. I’m Kazuhiro Yamashita from Kyushu University, Japan. Today, I would like to talk about my research. The title is “Thresholds for size and complexity metrics: A case study from the perspective of defect density.”
• #3: According to the survey by Emad Shihab, over 100 defect prediction papers have been published from 2000 to 2011. The primary goal of defect prediction is to provide guidelines to practitioners on what kind of code has better quality. More specifically, the papers want to find out: which metric is the most useful and what values of these metrics indicate good or bad code.
• #4: Previous papers claimed that extreme values of metrics are sign of poor quality. For example, McCabe claimed that complexity value 10 is the threshold from his experience. Other papers also showed relationships between metrics and quality.
• #5: However, Fenton found that previous papers have contradictory evidence. For instance, an early study argued that the number of defects increases with the number of code segments. Other studies showed that optimum size files have lower defect density. Also other studies confirmed larger modules have lower defect densities. So, the question is we can really use metric thresholds to identify risky files.
• #6: To solve the question, the goal of our study is to observe if a consistent relationship between metric thresholds and software quality. As the approach, we derive thresholds using a method proposed by Alves, then evaluate the thresholds using defects information.
• #7: In this study, we use a method proposed by Alves because that the method have following three characteristics. First, Data-driven. The method is driven by measurement data from a representative set of systems not by expert opinion Next, Robust. The method respects the statistical properties of the metric. Last, Pragmatic. The method is repeatable, transparent and straightforward to carry out.
• #8: This figure shows the overview of our approach. As we explained, our approach has two steps. First, we derive thresholds from sets of projects. Second, we evaluate the relationships between the thresholds and defects. Now we explain each part of our approach.
• #9: First we explain datasets.
• #10: In this study, we prepare four types of datasets to derive and evaluate thresholds. Because we want to evaluate relationship between thresholds and defects, we have to collect defect information for evaluation. But collecting defect information takes more time. Therefore, we prepare large sets of projects for derivation including only metrics information and small sets of projects for evaluation including metrics and defects information. For each of the purposes, we prepare two types of projects. One is from industry the other one is from open source software. The industrial datasets are provided by Avaya laboratory. The OSS projects are obtained from some forges like sourceForge and GitHub. Additionally, we collect the information of three projects Eclipse, Mylyn and Netbeans for evaluation.
• #11: One of the points of the method is using a representative set of systems. Since same company developed both sets of industrial projects for derivation and evaluation, projects for derivation would be a representative set of projects for evaluation. On the other hand, three OSS projects for evaluation are developed by different constitutions to the other OSS projects. Hence, we decide to select 1,000 representative projects from the 4,575 projects for each projects using median LOC because that we assume that representative means having a similar distribution of file sizes.
• #12: Next, we show details of Alves’ method.
• #13: Alves’ method consists of these 6 steps. Now we describe each step using an example.
• #14: Project A consists of N files and the total LOC is 10K. In this step, we calculate metrics for each file. For instance, cyclomatic complexity value of file 1 is 5 and that of file 2 is 1.
• #15: In addition to cyclomatic complexity, we also consider three metrics: Lines of Code, Module Interface size and Module Inward Coupling.
• #16: Next, we calculate weights of each file. In this example, total LOC of project A is 10K and LOC of file 1 is 300. Hence, the weight ratio of file 1 is 3%.
• #17: Now we obtained metric values and weight ratios of each file. In this step, we calculate weights of each metric value. In this example, only file 2 has cyclomatic complexity value 1, so the aggregated weight of cyclomatic complexity value 1 is 1%. Since file 1 and file 3 have cyclomatic complexity value 5, the aggregated weight is 8%. Like this, we calculate the aggregated weights for each metric value.
• #18: We perform the steps not only for project A but also for the set of projects. Now we obtained weights of each metric value per projects. Next we aggregate the weights within the set of projects. Then we calculate normalized weights of the set of projects by dividing by the number of projects.
• #19: Using the values of normalized weights, we calculate cumulative weight ratio, then plot the values.
• #20: Finally, we extract 70%, 80% and 90% values as thresholds. In this example, 5, 8 and 10 cyclomatic complexity values are thresholds. According to the thresholds, we can classify files into four categories: low risk, medium risk, high risk and very high risk.
• #21: Using Alves’ method, we obtained thresholds.
• #22: This table shows thresholds of cyclomatic complexity. From the table, all OSS projects have similar threshold values, but industrial projects have smaller values than OSS ones. Like this table, we also obtained thresholds of other three metrics.
• #23: Finally, we evaluate the thresholds using defect information.
• #24: In evaluation, we classify files of test projects according to the derived thresholds. Then, we evaluate the relationships between metrics and quality using defect proneness and density.
• #25: First we show the results with defect proneness.
• #26: These graphs show the results of industrial projects. First, we explain how to read the graphs. This graph shows four risk categories and the values of defect proneness and density. In this graph, blue dotted line shows LOC, black dotted line shows Module interface size, red line shows Cyclomatic Complexity and green line shows Module Inward coupling. In these graphs, when we focus on low risk category, we observe that the files have low defect proneness on all metrics. Additionally, when we focus on very high risk, we find high defect proneness with module interface size.
• #27: These graphs show the results with OSS projects. We observe that most of the projects have monotone relationship between metrics and defect proneness except for these two projects.
• #28: Next we show the results with defect density.
• #29: These graphs show the result with industrial projects. From the figures, we observe very high risk files classified by module interface size have high defect density
• #30: These graphs show the results with OSS projects. In Eclipse and Mylyn, we observe that defect density monotonically decreases with size metrics. When we focus on complexity metrics, we observe that the lines shape inverse-U. In Netbeans 5.0, we observe the relationship between metrics and density is all over the place. In Netbeans 4.0 the lines shape rotated-z shape. So, we conclude that we do not observe consistent trends with defect density.
• #31: To quantify if the defect density is indeed higher for the very high risk, we model a poisson model. From the model, we observe that almost coefficients are negative values. From that we assume that the highest risk files have lower defect density. However, only results of LOC and Complexity are statistically significant.
• #32: In this slide, we summarize our findings with the patterns of relationship argued by previous papers. With regard to defect proneness, we observe monotonically increase relationship. In terms of defect density, we observe the phenomenon that larger file have lower defect density in OSS projects. On the other hand, in industrial projects, we observe that larger file have larger defect density with size metric. In addition to three types of relationships, we also observed the relationship that medium file have larger defect density with complexity metrics.
• #33: From our study, we obtained the lessons learned. With regard to defect proneness, thresholds can identify defect prone files via size metrics. On the other hand, in defect density, since we did not find consistent trends, practitioners must proceed with caution when they use thresholds.
• #34: Now we conclude our slide. In this study, we aim to observe if a consistent relationship between metrics thresholds and software quality. For the goal, we derived thresholds using Alves’ method then evaluate the thresholds. From the results, we observed monotone relationship between metrics and defect proneness. However, with regard to defect density, we did not observe consistent relationship. That’s all, thank you.
• #36: This table shows thresholds of Lines of Code. From the table, all projects have similar threshold values. 70% is around 300 LOC. 80% is around 500. 90% is around 900. Especially, within same projects, they have almost same threshold if the version is different. But, only industrial projects have much larger value 1400. Like this table, we also obtained thresholds of other three metrics.
• #40: In the following slides, we show graphs like this. This graph shows four risk categories and the value of defect proneness and density. In this graph, blue dotted line shows LOC, black dotted line with circle shows Module interface size, red line shows Cyclomatic Complexity and green line with circle shows Module Inward coupling.
• #41: ここで，図表の読み方の説明 These graphs show the results of industrial projects. In these graphs, when we focus on low risk category, we observe that the files have low defect proneness on all metrics. Additionally, when we focus on very high risk, we observe high defect proneness with module interface size.