Analysis of Test Coverage ResultsTo achieve quality software in a cost-effective manner, a testing regimen typically consists of some combination of testing techniques. Static testing is used for checking such items as variable initialization, array boundaries, and null pointer usage. Dynamic testing is necessary to verify that overall the software behaves as expected, and exposes specific problems such as exceptions, memory leaks and performance bottlenecks. While the database underpinning Imagix 4D's analysis and visualization functionality is built through static analysis of source code, the resulting structural knowledge is also useful for leveraging dynamic test activities. In particular, Imagix 4D together with a dynamic test coverage tool can be used to evaluate whether the test cases being run have resulted in sufficient testing. The run time results generated by the test coverage tool can be imported and then viewed within Imagix 4D's overall structural context of the source code. This contextualization of the dynamic test results speeds identification of where further test cases are appropriate, and also helps with the analysis of performance issues. Testwell CTC++ from Verifysoft Technology GmbH, is the leading test coverage tool for embedded software. Through an integration with Testwell CTC++, Imagix 4D is able to aid in review of the test coverage data to speed the development of test cases and analysis of performance bottlenecks.
In an ideal world, test cases would be designed to achieve 100% coverage - execution of all paths through the code under all conditions. In a cost-effective world, test cases focus on exercising the more critical portions of the code more completely. Identifying which specific areas in the code need more testing, and then developing those test cases, is a job of the test engineer. To aid with this, the starting point typically would be the display of Testwell CTC++ results in Imagix 4D's Metrics tool. Three metrics are extracted from the dynamic test results and quantified at the function level - test coverage, execution frequency and execution time. In addition to these, the Metrics tool collects and displays over 20 other metrics. Several of these, such as McCabe Cyclomatic Complexity, Global Variables Used and Transitive Fan-In, provide an indication of how critical and error-prone a function might be. By seeing and sorting the test criticality metrics side-by-side with the test coverage metrics, initial decisions can be made about where further test case development might be targeted.
Analysis of Performance IssuesIn addition to reviewing and enhancing test coverage, the integration of Testwell CTC++ and Imagix 4D can be used for analyzing performance bottlenecks. For this, the function level execution frequency and time metrics are central. The starting point for this analysis is once again the Metrics window, where initial candidates for review are identified. At this point, the Function Calls graph, set up to indicate execution frequency and time becomes especially useful. Specific paths and clusters of performance critical functions can be more quickly identified. Analysis can be further aided by viewing side-by-side graphs of the same portion of the code. For example, one could show frequency and the other could show time.