Software Product Measurement and Analysis in a Continuous Integration Environment

Editor's Notes

• #9: The "why?" is important mainly because it defines how data should be interpreted providing the basis for reusing measurement plans and procedures for future projects and activities [5]. Figure 3 shows in the GQ(I)M the levels that can be applied to products, processes and resources in a life cycle of a system.
• #10: Following the GQ(I)M suggestion from a specialized work using Goal-Driven Software Measurement, it was found that the use of GQ(I)M complements the planning steps of MA of a PA within the CMMi [8]. The GQ(I)M top-down approach was applied to plan the measurement and analysis described in this paper, as shown in Table 2. Within the scope of this paper some product metrics were selected that could be automatically extracted from source codes and unit tests, with fixed frequency, through a Continuous Integration environment
• #16: The metrics organization in a structured database has allowed different ways of metrics analysis. Some of them were connecting client software like MS Excel to metrics database, generating statistical graphics, or exploring PivotTables [3].
• #17: But features and structure of transactional databases are tuned to support operational tasks like maintaining operational data by using On-Line Transaction Processing – OLTP. This solution might not be suitable if there is a large load of data metrics being collected and stored from a lot of projects, as in a Software Factory. In this case, would be necessary a structured database for high-performance analysis like a Data Warehouse (DW). Here, an approach for quickly answering multidimensional user queries would be also important, which could be achieved by using On-Line Analytical Processing (OLAP).
• #18: Once software metrics were loaded into a multidimensional database like a DW, they could be analyzed by using OLAP with good performance. That happened mainly because in the ETL processing, fact tables of metric values were aggregated by each possible grouping with the dimension tables attributes, and then a Cube was generated. So, the user could quickly execute common OLAP operations over the Metrics Cube. Among some used operations were the Drill-Down/Up and Slice-And-Dice, allowing users to analyze information in a more or less detailed view, or to change/combine different dimensions. Figure 5 shows an example of an OLAP analysis over the software metrics Cube. The matrix shows in lines the date/time of automated CI builds executed over time, and in columns it shows OO class names. In this example, matrix cells represent the Cyclomatic Complexity and Nb Lines of Code (LOC) metrics. So it can be seen how these metrics evolve during time, for each class. In this case, it can be noticed also that the Cyclomatic Complexity is increasing with the insertion of LOC, as expected. The user can Drill Up to analyze metrics in a higher level, from Package and Week perspectives, for example, or can also Slice data, like just filtering classes from one specific assembly.
• #19: The GQ(I)M approach preaches the importance of Graphical Indicators that makes sense even for a non-specialist. Generally, Indicators are understandable in a higher level than metrics, and are calculated over one or more metrics. In this work, it was generated a set of Key Performance Indicators (KPI) [8] that helped answering the Questions listed in the Table 2. An example of these KPI for a specific project, considering the desired Goal for them, is presented in Figure 6. This kind of approach has allowed the Project Manager to analyze, considering a time frame, how maintainable, reusable, and testable was the developed source code.
• #20: It was also possible the generation of a Kiviat graphic to allow an integrated analysis of the metrics maximum (desired) value against the measured values. As shown in Table 3 and Figure 8, in the case study’s project, the maximum measure for Cyclomatic Complexity (CC) found in a class was 87, whilst the assumed maximum desired CC value for a class was 50. That depicts that there are still some room to go improving the source code and reducing its complexity.
• #21: Another kind of performed analysis was the generation of statistical graphics. Figure 7 presents how the Unit Test Coverage was reduced in the project over time. From that, it can be inferred that, initially some unit tests were developed, but as the LOC increased in the next days, few new unit tests were created to test the inserted new methods.