Me: “How many bugs did you find during SW Unit testing?"SW developer/Unit testing responsible: "None"
Me:" Were any bugs identified during system bench testing that required changes to your code?
SW developer: "Yes, a few"
Me: “Why didn't you find those bugs during Unit testing?"
SW developer: --Silence--
Me: “Are you sure you did an effective SW Unit testing?"
SW developer: "I am sure. My statement and branch coverage are 100% and all my Unit test cases have passed! "
Me: --Silence--
Whenever we interact with clients regarding their Unit testing practices, we are often surprised by how much Unit testing has lost its original purpose in today’s automotive software development environment.
Let’s start by re-emphasizing the real value that Unit testing brings to a software program.
The purpose of SW Unit testing is to verify whether a software Unit, in isolation, behaves as intended by its design and the software requirements allocated to it.
To verify whether a Software Unit is implemented according to its design, the expected result of each Unit test case must be derived from the design — not from the code itself.
For example, consider a simple function that adds two numbers:
#include <stdio.h>
int add(int a, int b)
{
return (a + b);
}
int main(void)
{
int num1 = 10;
int num2 = 20;
int result;
result = add(num1, num2);
printf("Sum = %d\n", result);
return (0);
}
The design intent of this function is simple: add two numbers and return their sum.
Now, let's assume this code was implemented incorrectly as follows:
int add(int a, int b)
{
return (a + a + b);
}
During Unit testing, the expected result for the “add” function should be derived from the design intent.
- Expected result for the test case is: 30
- Actual result based on code: 40
- Unit test case fails
What happens in today's reality
Software teams often jump directly into coding, because design activities are often considered documentation overhead that delays the SW delivery. In many cases, Unit design is created afterwards by reverse-engineering the code itself and looks a lot more like code in documentation format (e.g., flowcharts with several steps, where each step simply mirrors a line of code)
As a result, there are no truly independent expected results against which the Unit test cases can be verified. The expected results for the Unit test cases are derived from the implementation itself. Taking the previous example of the add function, this means that the Unit tester will expect the test case to return 40 (based on code) and not 30 (which was the design intent).
As a result, software Unit testing is reduced to a process activity focused mainly on achieving coverage metrics such as statement, branch, and MC/DC coverage targets. Add additional process activities on top (ASPICE, ISO 26262) and there is more time wasted to create Unit test specifications, traceability to design, test reports, test result summaries, etc. All of this effort — just to prove that the code behaves according to itself?
As we recently heard from an industry expert: whenever quantitative metrics dominate, teams gradually lose focus on qualitative engineering aspects!
With the massive lines of code present in today’s complex ECUs, is it really worth spending time and resources verifying code against itself?
How can we change this reality?
Never code directly. Start off with at least a simple design on a piece of paper. What is this software module expected to do? What are its requirements? What are the functions that it should implement? What is the high-level logic that each function will implement? This does not require a large, detailed design work product, plus it gives teams a clear direction on the design intent of each function and how to verify them independently of the code.
Quality teams, software architects, and technical experts have a major responsibility in resetting the engineering culture around Unit testing. Unit design is not merely documentation; It is an upfront engineering investment to reduce the overflow of software defects discovered during later system-level testing phases. The reviewer of the Unit testing should verify whether all expected test results were derived based on design or requirements and not based on code. A complete absence of defects found during Unit testing should itself trigger questions about the effectiveness of the process.
Comments
Post a Comment