Embedded applications code coverage

Introduction

It is very important to know the quality of test cases executed for your application before releasing your application. Applications should be tested in all possible scenarios using different test cases. For complete coverage of code, an engineer should analyze the code by running the application with test cases prepared based on SRS requirements. After running the application once, we get complete stats of how well our test cases are covering the code. And based on the results, we can add a few more test cases. Code coverage tools will track the lines that are covered when we run the targeted application on a platform. These tools will show some of the statistics like; branches covered, functions covered and lines covered in your applications. There are different ways to setup the code-coverage for any application depending on the platform the application is running on.

For embedded Linux applications, we can use the following tools:

For windows applications, we have the following tools to analyze the code coverage

Code Coverage evaluation in Linux using Gcov

Gcov is a feature of GCC where it is enabled as a build option of GCC in the makefile or by providing some arguments to GCC build command.

Any type of C code with threads, daemonization applications etc, can be evaluated with Gcov option. In most cases, we don’t need to modify the source code for generating code-coverage related statistics, but there are a few cases which need an addition of a line of code before exiting the application i.e. just before return call of the main function. For example, while conducting a code-coverage analysis on daemon applications where it may have parent and child process and one process forks the other process to make the 2nd one as a child, we can expect code coverage for only the parent process. To get the coverage of child process too, we need to add “__gcov_flush(void);” in the application before the return call of child application.

Enabling Gcov code coverage for Linux C application

While writing any c application in Linux it needs to be built with GCC command. Assume that the source code file name is 'test.c'. To build this file, we need to run command 'GCC test.c -o test” and to enable the code coverage we need to run 'GCC test.c -o -g -O0 --coverage test' command. After running this command we get object files for c files and temporary files related to code coverage with extension gcno. After generating the binary, we can run this binary, so that it generates more temporary files with extension gcda. These files are related to code coverage. We need to place gcno, gcda and Object files related to the source code in the same folder and run the following commands in the same folder.
After running the above 2 commands we see many temporary files related to code coverage being generated. Out of all the files, index.html will have the details related to code coverage.

Following are the stats available in the HTML file:

Source file names are displayed as links which show the source code upon clicking. This way we can easily see the lines, branches, and functions which are not covered. Inside each source file, a link named 'functions' is present at the top. Clicking on that link shows the list of functions that are defined in that particular source file and also information related to how many times that function is called. Every source file has a number displayed beside each line which is the hit count that tells how many times that particular line has been executed.

Understanding branches in code-coverage using Gcov

One can easily write test cases covering lines and functions without much effort. We can easily cover at least 85 to 90% of functions and lines in any kind of application, but in order to cover branches, some level of understanding related to Gcov is required while writing test cases.

Example for branch statement [1]:
if(test_variable == 1)
{
printf(“Variable == 1”);
}
else
{
printf(“Variable != 1”);
}

In this case, If test_variable is 1, then we can cover 2 lines and 1 branch, which results in uncovered lines = 2 and uncovered branches = 1, i.e., else statement.

Assume that our code has only if statement, in this case there won’t be any uncovered lines but there will be uncovered branches in results.
Example branch statement [2]:
if(test_variable == 1)
{
printf(“Variable == 1”);
}
In the above example, if we write a test case which makes test_variable to 1 then
Not only the ‘if else’ statements, but the branch stats also include details of decision based code that are involved in while loops, for loops, switch case statements etc.

Barriers for 100% code coverage

Branches which depend on return values of some library function will restrict the 100% code coverage because we don’t have control on most of the system functions return values.

Steps to follow for better coverage

To take control of each and every function call, an extra piece of code should be developed by the developer and added, so that code will behave according to your requirement. These are the design level rules a developer should follow during development stage for better coverage.

1. System call return value modification
Example:
FILE *fp = NULL;
fp = fopen(“/mnt/mount_dev/test_file.txt”, “w”);
if(fp == NULL)
{
printf(“file open failed”);
}

In this case to cover fp == NULL scenario we need to delete “/mnt/mount_dev” folder, but assume that “/mnt/mount_dev” is a mount point of an external storage device, in such case if you want to make fp == NULL(without unmounting the external storage device), you need to overwrite fp value to NULL. Below is a sample code for assigning fp with NULL.
//LCOV_EXCL_START
#ifdef SYS_TEST
{
FILE *testfp = NULL;
testfp = fopen("/home/root/fp_null.txt", "r");
if(testfp != NULL)
{
fp = NULL;
fclose(testfp);
system("rm -rf /opt/root/fp_null.txt");
}
}
#endif
//LCOV_EXCL_STOP
This code will make fp to NULL if /home/root/fp_null.txt file is available", add this code after “fp = fopen(“/mnt/mount_dev/test_file.txt”, “w”);” instruction, build the code and run the test again.

2. Switch case statement changes
Example:
switch(var)
{
Case 1: printf(“var = 1”);
break;
Case 2: printf(“var = 2”);
break;
Case 3: printf(“var = 3”);
break;
}
In this case, to cover default statement of switch condition (even when the default statement is not present in the code, gcov will look for failed condition of Case 3 and show as 1 uncovered branch if “1 <= var <= 3”) we need to have test case which assigns var with a value which is < 1 or > 3. If we don't have any control on the var variable, we need to overwrite var contents with 4 or 0, so that we can cover the default statement too. A sample code which modifies var value is shown below:
//LCOV_EXCL_START
#ifdef SYS_TEST
{
FILE *testfp = NULL;
testfp = fopen("/home/root/change_var.txt", "r");
if(testfp != NULL)
{
var = 4;
fclose(testfp);
system("rm -rf /opt/root/change_var.txt");
}
}
#endif
//LCOV_EXCL_STOP

Add this code before “switch(var)” line, build the code, and run the test again. This code will make var value 4 if we create a file at run time /home/root/change_var.txt.

3. Handling return value of library functions used in our application
Example:
function1()
{
int ret = function2();
if(ret == 0)
{
printf(“return value is 0”);
}
}
In this case, function2 () is a user defined function (which will always return 0) but is defined in another other library. Without knowing the function2 () behavior, it is not possible to make ret value other than 0, the best way is to change ret value at run time as shown below:
//LCOV_EXCL_START
#ifdef SYS_TEST
{
FILE *testfp = NULL;
testfp = fopen("/home/root/change_ret.txt", "r");
if(testfp != NULL)
{
ret = 1;
fclose(testfp);
system("rm -rf /opt/root/change_ret.txt");
}
}
#endif
//LCOV_EXCL_STOP

Add this code after “int ret = function2();” instruction, build the code, and run the test again. This code will make var value 4 if we create file at run time /home/root/change_var.txt.
Any code between //LCOV_EXCL_START and //LCOV_EXCL_STOP statements will be neglected by Gcov tool.

Sample code with if else statements for understanding the 100% code – coverage

Source Code
//File name: test_branches.c
#include
#include
int main(int argc, char *argv[])
{
if(argc != 7)
return 1;
/***********************AND******************************/
if((atoi(argv[1]) == 1) && (atoi(argv[2]) == 4))
{
printf("\nTest case1\n");
}
/***********************OR*******************************/
if((atoi(argv[3]) == 2) || (atoi(argv[4]) == 8))
{
printf("\nTest case2\n");
}
/***********************IF and Else_IF*******************/
if(atoi(argv[5]) == 16)
{
printf("\nTest case3\n");
}
else if(atoi(argv[5]) == 8)
{
printf("\nTest case4\n");
}
else if(atoi(argv[5]) == 4)
{
printf("\nTest case5\n");
}
/***********************Only If***************************/
if(atoi(argv[6]) == 2)
{
printf("\nTest case6\n");
}
return 0;
}


Build command
gcc -O3 test_branches.c -Wall -fprofile-arcs -ftest-coverage -o test_branches