Embedded applications code coverage

January 17, 2017

Introduction

It is very important to know the quality of test cases executed for your application before releasing your application. Applications should be tested in all possible scenarios using different test cases. For complete coverage of code, an engineer should analyze the code by running the application with test cases prepared based on SRS requirements. After running the application once, we get complete stats of how well our test cases are covering the code. And based on the results, we can add a few more test cases. Code coverage tools will track the lines that are covered when we run the targeted application on a platform. These tools will show some of the statistics like; branches covered, functions covered and lines covered in your applications. There are different ways to setup the code-coverage for any application depending on the platform the application is running on.
For embedded Linux applications, we can use the following tools:
1. Squish Coco
2. CoverageMeter
3. Gcov etc.
For windows applications, we have the following tools to analyze the code coverage
1. xCover
2. CoverageMeter
3. BullseyeCoverage
4. C++ code coverage

Code Coverage evaluation in Linux using Gcov

Gcov is a feature of GCC where it is enabled as a build option of GCC in the makefile or by providing some arguments to GCC build command.
Any type of C code with threads, daemonization applications etc, can be evaluated with Gcov option. In most cases, we don’t need to modify the source code for generating code-coverage related statistics, but there are a few cases which need an addition of a line of code before exiting the application i.e. just before return call of the main function. For example, while conducting a code-coverage analysis on daemon applications where it may have parent and child process and one process forks the other process to make the 2nd one as a child, we can expect code coverage for only the parent process. To get the coverage of child process too, we need to add “__gcov_flush(void);” in the application before the return call of child application.

Enabling Gcov code coverage for Linux C application

While writing any c application in Linux it needs to be built with GCC command. Assume that the source code file name is 'test.c'. To build this file, we need to run command 'GCC test.c -o test” and to enable the code coverage we need to run 'GCC test.c -o -g -O0 --coverage test' command. After running this command we get object files for c files and temporary files related to code coverage with extension gcno. After generating the binary, we can run this binary, so that it generates more temporary files with extension gcda. These files are related to code coverage. We need to place gcno, gcda and Object files related to the source code in the same folder and run the following commands in the same folder.
lcov -directory . -c -o test.info
genhtml test.info
After running the above 2 commands we see many temporary files related to code coverage being generated. Out of all the files, index.html will have the details related to code coverage.
Following are the stats available in the HTML file:
• Line coverage stats in each file
• Functions coverage in each file
• Branch coverage in each file
• Total lines covered in all source files
• Total branches covered in all source files
• Total functions covered in all source files
• In individual source files, for each line, hit count is maintained just beside that particular line
Source file names are displayed as links which show the source code upon clicking. This way we can easily see the lines, branches, and functions which are not covered. Inside each source file, a link named 'functions' is present at the top. Clicking on that link shows the list of functions that are defined in that particular source file and also information related to how many times that function is called. Every source file has a number displayed beside each line which is the hit count that tells how many times that particular line has been executed.

Understanding branches in code-coverage using Gcov

One can easily write test cases covering lines and functions without much effort. We can easily cover at least 85 to 90% of functions and lines in any kind of application, but in order to cover branches, some level of understanding related to Gcov is required while writing test cases.
If and else branch statements:
In case of if, elseif and else statements, we need to know that for if there should always be an else statement. When we do not write an else statement, gcov will assume some other else statement and shows it for that particular if.

Example for branch statement [1]:
if(test_variable == 1)
{
printf(“Variable == 1”);
}
else
{
printf(“Variable != 1”);
}

In this case, If test_variable is 1, then we can cover 2 lines and 1 branch, which results in uncovered lines = 2 and uncovered branches = 1, i.e., else statement.
Assume that our code has only if statement, in this case there won’t be any uncovered lines but there will be uncovered branches in results.
Example branch statement [2]:
if(test_variable == 1)
{
printf(“Variable == 1”);
}

In the above example, if we write a test case which makes test_variable to 1 then
Not only the ‘if else’ statements, but the branch stats also include details of decision based code that are involved in while loops, for loops, switch case statements etc.

Barriers for 100% code coverage

Branches which depend on return values of some library function will restrict the 100% code coverage because we don’t have control on most of the system functions return values.

Steps to follow for better coverage

To take control of each and every function call, an extra piece of code should be developed by the developer and added, so that code will behave according to your requirement. These are the design level rules a developer should follow during development stage for better coverage.
1. System call return value modification
2. Switch case statement changes
3. Handling return value of library functions used in the application
1. System call return value modification
Example:
FILE *fp = NULL;
fp = fopen(“/mnt/mount_dev/test_file.txt”, “w”);
if(fp == NULL)
{
printf(“file open failed”);
}
In this case to cover fp == NULL scenario we need to delete “/mnt/mount_dev” folder, but assume that “/mnt/mount_dev” is a mount point of an external storage device, in such case if you want to make fp == NULL(without unmounting the external storage device), you need to overwrite fp value to NULL. Below is a sample code for assigning fp with NULL.
//LCOV_EXCL_START
#ifdef SYS_TEST
{
FILE *testfp = NULL;
testfp = fopen("/home/root/fp_null.txt", "r");
if(testfp != NULL)
{
fp = NULL;
fclose(testfp);
system("rm -rf /opt/root/fp_null.txt");
}
}
#endif
//LCOV_EXCL_STOP

This code will make fp to NULL if /home/root/fp_null.txt file is available", add this code after “fp = fopen(“/mnt/mount_dev/test_file.txt”, “w”);” instruction, build the code and run the test again.
2. Switch case statement changes
Example:
switch(var)
{
Case 1: printf(“var = 1”);
break;
Case 2: printf(“var = 2”);
break;
Case 3: printf(“var = 3”);
break;
}

In this case, to cover default statement of switch condition (even when the default statement is not present in the code, gcov will look for failed condition of Case 3 and show as 1 uncovered branch if “1 <= var <= 3”) we need to have test case which assigns var with a value which is < 1 or > 3. If we don't have any control on the var variable, we need to overwrite var contents with 4 or 0, so that we can cover the default statement too. A sample code which modifies var value is shown below:
//LCOV_EXCL_START
#ifdef SYS_TEST
{
FILE *testfp = NULL;
testfp = fopen("/home/root/change_var.txt", "r");
if(testfp != NULL)
{
var = 4;
fclose(testfp);
system("rm -rf /opt/root/change_var.txt");
}
}
#endif
//LCOV_EXCL_STOP

Add this code before “switch(var)” line, build the code, and run the test again. This code will make var value 4 if we create a file at run time /home/root/change_var.txt.
3. Handling return value of library functions used in our application
Example:
function1()
{
int ret = function2();
if(ret == 0)
{
printf(“return value is 0”);
}
}

In this case, function2 () is a user defined function (which will always return 0) but is defined in another other library. Without knowing the function2 () behavior, it is not possible to make ret value other than 0, the best way is to change ret value at run time as shown below:
//LCOV_EXCL_START
#ifdef SYS_TEST
{
FILE *testfp = NULL;
testfp = fopen("/home/root/change_ret.txt", "r");
if(testfp != NULL)
{
ret = 1;
fclose(testfp);
system("rm -rf /opt/root/change_ret.txt");
}
}
#endif
//LCOV_EXCL_STOP

Add this code after “int ret = function2();” instruction, build the code, and run the test again. This code will make var value 4 if we create file at run time /home/root/change_var.txt.
Any code between //LCOV_EXCL_START and //LCOV_EXCL_STOP statements will be neglected by Gcov tool.

Sample code with if else statements for understanding the 100% code – coverage

Source Code
//File name: test_branches.c
#include
#include
int main(int argc, char *argv[])
{
if(argc != 7)
return 1;
/***********************AND******************************/
if((atoi(argv[1]) == 1) && (atoi(argv[2]) == 4))
{
printf("\nTest case1\n");
}
/***********************OR*******************************/
if((atoi(argv[3]) == 2) || (atoi(argv[4]) == 8))
{
printf("\nTest case2\n");
}
/***********************IF and Else_IF*******************/
if(atoi(argv[5]) == 16)
{
printf("\nTest case3\n");
}
else if(atoi(argv[5]) == 8)
{
printf("\nTest case4\n");
}
else if(atoi(argv[5]) == 4)
{
printf("\nTest case5\n");
}
/***********************Only If***************************/
if(atoi(argv[6]) == 2)
{
printf("\nTest case6\n");
}
return 0;
}
Build command
gcc -O3 test_branches.c -Wall -fprofile-arcs -ftest-coverage -o test_branches

Test cases and its commands to be executed to get 100% coverage of branches and lines

First we focus on covering 100% lines (using first 4 test cases mentioned below) and then we can head over to the uncovered branches
Test case 1:/test_branches
This test case will cover the fail make “argc != 7” condition TRUE and covers first branch in the source code.
Gcov html report after first test case execution
Figure 1: Gcov html report after first test case execution
Test case 2: ./test_branches 1 4 2 8 16 2
First test case has covered the first if statement and after which the application will exit. To cover remaining part of the code we need to provide 7 arguments to application and this “test case 2” will focus on covering remaining 2 if statements by setting argv [3] and argv[6] to value = 2.
Gcov html report after second test case execution
Figure 2: Gcov html report after second test case execution
Test case 3: ./test_branches 2 4 1 8 8 1
This test case will focus on covering “else if (atoi(argv[5]) == 8)” statement and else statement of “if(atoi(argv[6]) == 2)” statement. For this, argv[5] should be equal to 8 and argv[6] should be not be equal to 2.
Gcov html report after third test case execution
Figure 3: Gcov html report after third test case execution
Test case 4: ./test_branches 1 4 2 8 4 2
After execution of test case 3, in report we can see only one line not being covered which depends on argv[5] value. If argv[5] is equal to 4 then the remaining line will be covered with this test case.
Gcov html report after fourth test case execution
Figure 4: Gcov html report after fourth test case execution
Test case 5:/test_branches 1 1 1 1 2 2
After execution of test cases 1, 2 3 and 4, we can see 100% of the lines are covered in the report, but still some branches are left out. We know that line and branch coverage can be independent in special cases. This is one of such case which is why we are there is less branch coverage even with 100% line coverage. If we analyze the logical statements which gcov has not covered, we can easily write test case for that scenario. First branch which was not covered is “if((atoi(argv[1]) == 1) && (atoi(argv[2]) == 4))”, this statement has and operation where if one condition fails, it’s enough to skip the code inside if. But if you see all 4 test cases, we don’t have test case which fails (atoi(argv[2]) == 4) condition. Second branch which is not covered is “else if(atoi(argv[5]) == 4)”. This statement is expecting a test case which makes with argv[5] != 4, but argv[5] should not be 8 or 16 because if it 16 or 8 then the test case will cover the previously covered statements “else if(atoi(argv[5]) == 8)” and “if(atoi(argv[5]) == 16)”.
Gcov html report after fifthtest case execution
Figure 5: Gcov html report after fifth test case execution
In gcov code-coverage reports, following are the color and symbol indications:
For source code line:
• Light blue: indicates line is covered during the execution.
• Light red color: Line is not covered during execution.
• + symbol beside branch condition: condition has hit during the execution
• - symbol beside branch condition: condition was not hit during the execution.
Coverage percentage column of complete summary stats on top right (check any of the model report mentioned above for 5 test case)
• Light green color: For more than 85% coverage
• Light Yellow color: For 75 to 85%.
• Red color: For less than 75% coverage.

Conclusion

All code coverage tools are developed for only one purpose i.e to improve the quality of validation by providing different statistics, in all code coverage tools. Common and basic stats that will be displayed are lines, branches and functions covered by calling them in the application. These stats will be enough for any validator for improving the quality of test cases. Of all code-coverage tools, Gcov is the best tool for Linux Platforms which is available for free and provides sufficient information related coverage. Compared to other tools, setting up of gcov is easy which just needs addition of some flags to build command. For windows application, the best tool is “C++ Coverage Validator”.