Software Quality Analysis with Alitheia Core

Editor's Notes

• #2: \n
• #3: \n
• #4: \n
• #5: \n
• #6: \n
• #7: \n
• #8: \n
• #9: \n
• #10: \n
• #11: \n
• #12: \n
• #13: \n
• #14: In large organisations, software quality evaluations are performed by dedicated teams that assess a project’s conformance to the specifications\n
• #15: Ενώ σε άλλες εμπειρικές επιστήμες το να βρούμε δεδομένα είναι ακριβή διαδικασία \nπχ στην Ιατρική ...., στη φυσική CERN...\nΣτον κλάδο μας έχουμε πολλά υψηλής ποιότητας εμπειρικά δεδομένα OSS. Σε σύνολο ο Mockus σε πολύ πρόσφατη δημοσίευση αναφέρει τον όγκο τους >ΤΒ και το σύνολο των έργων σε >250κ\n
• #16: This may sound like a rhetorical question, but it is not. \n
• #17: Then there is the problem of format disparity. Most research so far was done with CVS and Bugzilla. In reality, there are a lot of tools that store process data \n
• #18: Data sizes for large experiments are very large\n
• #19: All those reasons led our research group to the SQO-OSS project, which produced the Alitheia Core tool. The project’s aim was to produce software quality analysis tools, but the original targets strayed towards creating infrastructures rather than the tools themselves\n
• #20: All those reasons led our research group to the SQO-OSS project, which produced the Alitheia Core tool. The project’s aim was to produce software quality analysis tools, but the original targets strayed towards creating infrastructures rather than the tools themselves\n
• #21: Alitheia Core is a resear metric plugin-based architecture and a corresponding engine\n
• #22: new product and process software metrics that take advantage of this infrastructure\n
• #23: \n
• #24: \n
• #25: as you can see …\n\nthin brings concrete objects\nfat combines multiple elements (metadata with data, and from multiple data sources)\n
• #26: Here is how the most important parts work together. The project data is maintained on a project mirror, which is managed externally though a set of perl scripts. The metadata updater component uses this data to extract metadata and put it in the DB. Metrics use higher level services that combine raw data and metadata to get their input and store directly in the DB\n
• #27: What alitheia core tries to do is to maintain a database of metadata and metric results. \nThe schema is hierarchical and everything is bound to a project. \n \n
• #28: The metadata update process is a two stage process\n
• #29: In the first stage the system gets raw data (source code logs, mail messages, and bug reports) and extracts just enough meta data to create abstract representations in the database. No real data goes in the database, just metadata.\n
• #30: In the second stage, the metadata are used to extract semantic relationships among metadata entities. For example, to extract bug numbers from commit messages and create links between the to or to recover developer identities or to reconstruct mailing list threads\n
• #31: \n
• #32: A metric defines an activation type which is an entity in the metadata schema I have presented earlier and also declares depencencies to other metrics. After that the system will automatically invoke the metric when it finds new instances of the activation type after an update. Let’s see an example\n
• #33: \n
• #34: This is the implementation of the line counting plug-in I ‘ve presented you earlier on, minus some bureaucracy (constructors, imports etc). Just about 20 Locs to retrieve a file from a revision, read its lines and store results in a database. This is comparable to a shell or python script, but it is way much faster. The abstractions Alitheia Core are very high level and cross platform and this is why the overhead it adds in algorithm implementation is minimal\n
• #35: \n
• #36: \n
• #37: \n
• #38: Let us first walk through the life-cycle of a metric plugin\n
• #39: Specifically, let’s now see how we can calculate a metric (wc in this case) for a simple file\n
• #40: Updater sees that a file has changed for a given method and calls the run method for the LoC metric\n
• #41: and later on the scheduler runs the job\n
• #42: The job obtains the details about the object (file, mailing list, bug) it has to calculate \n
• #43: and updates the database with the result\n
• #44: and now let us take a walkthrough of the corresponding client\n
• #45: Client asks a metric for a given file version\n
• #46: Web service calls the plugin administrator to obtain a reference for the specific plugin\n
• #47: Given the reference in can then call the metric\n
• #48: The metric accesses the database to obtain the results in a metric-native format\n
• #49: and the web service returns the results to the client\n
• #50: SQO-OSS is designed to handle hundreds of projects, tens of thousands of files, hundreds thousands of revisions, messages, and bug reports.\nThis means it is architected for performance.\nLet me give you an example\n
• #51: A very crude example is the retrieval of metadata for the live files in a particular version. In Alitheia Core this a database query which executes in about 2 secs, even under load\n
• #52: The corresponding time for checking out XBMC. This small example shows how the metadata storage in the database pays off\n
• #53: The workloads we are processing are embarrassingly parallel due to the fact that projects are independent from each other.\n The execution path from the mirror to the metadata database is lock free. Only plug-ins that process data across projects need to be careful when updating a shared field in the DB. This allow us to scale almost for free.\nWe implemented a very simple clustering system that binds a project to the node it was registered from and does not allow any other type of processing to take place on other nodes. \n
• #54: This is our cluster: Left a 3 thread processing node + project server, middle a 6 thread processing core, bottom mirroring and storage of raw data, file server, database and 8 processing threads to keep the CPUs busy while I/O, right web server + 16 slow processing threads. To scale, we just need more processing nodes.\n
• #55: Given enough data to process, this simple setup can drive all machines involved to their knees. For example this screenshot displays info from the db server while all 3 machines (node, web server, db server) run metrics. 2.35k queries per second, 46 open transactions, 32 active jobs on the web server. All three machines combined can eat up more than 500 wc jobs per second or 50 contribution plugin jobs per second.\n
• #56: Another example of the cluster on its knees - Linear scalability display: The screen of the database server running just the database while other nodes start connecting: Queries Per Second increase is almost linear and load equals to processor cores -> the machine is saturated. \n
• #57: \n
• #58: The SQO-OSS web site or the Eclipse plug-in display metrics on various quality metrics immediately after each commit! For example, imagine you are writting some code and when you commit you immediately see its effects on various metrics (e.g. Coupling, dependencies) on the whole system! Or when you do a montly code inspection to be able to immediately recognise bad code smells....\n
• #59: .... And be automatically notified when bad code smells start to build up or on the other hand when the system is doing well and is probably ready for release\n
• #60: Other possibilities include: test case integration, finding classes with no test cases, integration of various custom-developed tools for monitoring progress. \n\n
• #61: SQO-OSS is platform not just a web site. It was build to allow easy integration of tools, already have C++ and Python based interfaces. One can write plug-ins in either Java or C++ or Python. In fact we have wc in 3 languages :-) A text output parser plug-in is possible by executing the program in C++ or Java.\n
• #62: Convenient access to data, Data preprocessed in relational format. DB dumps will be made available from our web site. \n
• #63: \n