Analysis of Test Coverage Results

To achieve quality software in a cost-effective manner, a testing regimen typically consists of some combination of testing techniques. Static testing is used for checking such items as variable initialization, array boundaries, and null pointer usage. Dynamic testing is necessary to verify that overall the software behaves as expected, and exposes specific problems such as exceptions, memory leaks and performance bottlenecks.

While the database underpinning Imagix 4D's analysis and visualization functionality is built through static analysis of source code, the resulting structural knowledge is also useful for leveraging dynamic test activities. In particular, Imagix 4D together with a dynamic test coverage tool can be used to evaluate whether the test cases being run have resulted in sufficient testing. The run time results generated by the test coverage tool can be imported and then viewed within Imagix 4D's overall structural context of the source code. This contextualization of the dynamic test results speeds identification of where further test cases are appropriate, and also helps with the analysis of performance issues.

Testwell CTC++ from Verifysoft Technology GmbH, is the leading test coverage tool for embedded software. Through an integration with Testwell CTC++, Imagix 4D is able to aid in review of the test coverage data to speed the development of test cases and analysis of performance bottlenecks.
In an ideal world, test cases would be designed to achieve 100% coverage - execution of all paths through the code under all conditions. In a cost-effective world, test cases focus on exercising the more critical portions of the code more completely. Identifying which specific areas in the code need more testing, and then developing those test cases, is a job of the test engineer.

To aid with this, the starting point typically would be the display of Testwell CTC++ results in Imagix 4D's Metrics tool. Three metrics are extracted from the dynamic test results and quantified at the function level - test coverage, execution frequency and execution time. In addition to these, the Metrics tool collects and displays over 20 other metrics. Several of these, such as McCabe Cyclomatic Complexity, Global Variables Used and Transitive Fan-In, provide an indication of how critical and error-prone a function might be. By seeing and sorting the test criticality metrics side-by-side with the test coverage metrics, initial decisions can be made about where further test case development might be targeted.

In the Function Metrics display, you control which metrics are displayed and how they are sorted.

Once initial targets have been identified, the next step could be to review those functions in a Graph window. Normally, the Function Calls graph is used to study the control flow surrounding some set of functions or implementing some functionality in the software. Through the Testwell CTC++ integration, these graphs can be augmented to indicate the test coverage results. In the resulting displays, you're able to identify areas where several functions with limited coverage are clustered, often a sign that additional testing is appropriate.

In the Function Calls graph, the function rectangles are colored to indicate test coverage. This is either relative to the other visible functions or to thresholds you establish. Either way, the color range from green to red indicates the range from higher to lower coverage. Blue indicates library functions, where coverage data does not apply.

For reviewing the test coverage results of a specific function, the Flow Chart displays are used. The Flow Charts show the program logic / control flow within a given function. When augmented with the test coverage data, they indicate which portions of the logic / flow have not been exercised by the test cases. This information can be used to aid code reviews (ie, are there reasons that these paths are not being executed?) as well as to extend test cases.

In the Logic layout, the generally diamond-shaped symbols represent decisions, while blocks of in-line code are contained in rectangles. Unexercised code is indicated by red shapes and paths.

Several layouts are available for viewing the program flow. The Flow Chart above emphasizes to the program logic of the function, while the Flow Chart below indicates more the internal structure of function's control flow.

The Structure layout uses the same decision and code shapes as the Logic layout.

Analysis of Performance Issues

In addition to reviewing and enhancing test coverage, the integration of Testwell CTC++ and Imagix 4D can be used for analyzing performance bottlenecks. For this, the function level execution frequency and time metrics are central.

The starting point for this analysis is once again the Metrics window, where initial candidates for review are identified. At this point, the Function Calls graph, set up to indicate execution frequency and time becomes especially useful. Specific paths and clusters of performance critical functions can be more quickly identified. Analysis can be further aided by viewing side-by-side graphs of the same portion of the code. For example, one could show frequency and the other could show time.

In the Function Calls graph, colorizing can be based on any of the function level metrics, as is shown here for execution frequency.

Test Coverage and
Dynamic Analysis