What is code coverage?
Code Coverage (also known as Test Coverage) is the degree to which the code within a software code base has been tested. Generally expressed as a percentage, it is used to indicate the level of thoroughness that has been achieved whilst carrying out live testing of the compiled codebase. There are different granularities, or levels of coverage possible. For example, number of statements or functions covered. There are others that are sensitive to the program logic such as branch coverage, which looks to check whether all possible paths of execution have been exercised (e.g, following the true/false branches through an if statement), and condition coverage, which looks to verify that all logical permutations of boolean sub-expressions have been exercised (e.g, in the statement “if ((x > 0) && (x < 10)”, check the 4 logical possibilities for each of the two sub expressions have been exercised by the testing – false and false, false and true, true and false and true and true). This last form is a specialist of condition coverage called Multi Condition/Decision Coverage, or MC/DC.
Why analyse code coverage?
The obvious reason to understand your test coverage is to make sure the range of functionality of an application is thoroughly tested. Coverage analysis provides a feed back loop to the live testing phase, making sure the set of live tests being performed are complete, and maintain relevancy where the functionality of the application changes.
Monitoring test coverage as part of the development process that allows developers and managers to gain insight into the reach of the tests they are performing on the code they’re producing, and providing feedback on potential improvements on tests required.
Another key driver of adoption are the many development standards for the development in safety critical applications such as the Automotive industries ISO 26262 and the Aerospace industries DO-178B/C that recommend or require coverage analysis be performed in order to achieve certification.
Organisations that combine code coverage with regression testing and static analysis are more efficient and productive. Code coverage can help identify unused or dead code which can be removed from the footprint of the on-target code.
What to look for in a code coverage tool
All code coverage tools will allow you to perform host based testing of your code but for many applications it is important that the application code is tested on the target device. Many of the software development standards for safety critical applications mandate on-target code coverage as part of the certification process.
How does Code Coverage work
Code coverage works by instrumenting your source code. When the code is then compiled and executed on the host or target during testing or execution data is collected from the instrumentation and collected for processing by the code coverage tool.
Instrumentation is the act of adding code to a codebase, suitable to perform some activity that has no impact otherwise on the behaviour of the application. Typically, it is done by instrumenting the target codebase with probes, automatically inserted at key points within the codebase as required. For example, with function coverage, a probe will be added as the first statement in each function. At its simplest a probe is a boolean that is set true if the function is executed during testing.
Once the code base is instrumented the tests can be executed on the instrumented code base. This can occur on the host or on the target devices as requirements dictate. Once the test runs are complete the code coverage data can be exported for analysis.
At the point the live test ends, another section of embedded instrumentation code will export all the probe data that indicates which functions were executed (or not) so that you can report on the test coverage for the test run.