Understanding Software Testing Metrics
Understanding Software Testing Metrics
As QA professionals, we all know how invaluable metrics can be when determining the testing effectiveness of a system. Software systems can be delivered to production with or without metrics to support it’s quality or effectiveness of testing. There are hundreds of metrics that are available to the QA professional but what metrics provide the most value to the organization? But, let’s get a few things known first.
What are Software Testing Metrics?
Software testing metrics are a universal way to monitor, measure and provide visibility into the testing activities before, during and after the testing period. Metrics are a way for organizations to gage the success of not just the testing, but development, project management, teamwork, and the quality of the software system. This is vehemently a strong reason why organizations ask for metrics, even when they don’t understand the metrics being presented.
Metrics used in any organization for a system should make sense. Every organization’s testing team is going to have a different opinion regarding what metrics to be used. A metric conveys the thought or the result of a combination of data that is collected during the testing. An average sized project, could use for example, 24 metrics while a large project may use only 12. Why such an odd comparison? It’s simple. Metrics provided are not based on the size of the project but on the type of project communication that the team wants to show the stakeholders. In other words, metrics must make sense!
QA professionals should not provide every metric or metrics that have no relevance in the project. Just like everything else, metrics have to make sense.
What test metrics should be used?
Organizations realize that there are hundreds of metrics in the testing world today but in reality, there is only a handful to be used. That puts the burden on the testing team to determine the metrics but with a little help, it is an easy task. Fundamental metrics are all about the absolute numbers. These numbers, such as the total number of test cases used, the number of test cases passed, the number of test cases failed, and so forth are a starting point. But they can’t exist alone. These absolute numbers produce derivative numbers, which are much more important than people realize.
Derivative metrics provide organizations with the information needed to measure quality, reliability and effectiveness before migrating their system to production. Derivative metrics also provide sanity for the reliance of the overall customer experience.
Some examples of metrics that organizations should really consider are
- Fixed Defects Percentage
- Defects Rejected
- Average Time for Development to Fix Defects
- Test Efficiency
- Defect Rate per Testing Hour
- Requirement Defect Density
How Do I Choose What Metrics To Use?
There is no right or wrong answer here. Organizations and the testing team need to determine what metrics make sense to all. Not every organization needs to have every single metric or the few examples listed above. Instead, the testing team needs to decide with the rest of the project team, what makes sense. For example, does the business or the customer care about how many defects were actually found during testing? Or do they care more about how many defects were found as a result of ambiguous and poorly written requirements?
Metrics are a combination of being both subjective and objective.
Organizations can go overboard with metrics. It is important to remember that producing metrics and providing tables, charts, etc take time and that time is part of the testing planning and execution timeline. It needs to make sense.
Metrics are used to tell the story of the system that is under test. That story is what stakeholders, customers, organizations and project team members care about. That story is what will help the success of the project as well as aiding in providing customer satisfaction. As QA professionals, remember to pick and choose the metrics that make sense to tell the story.