What Is Requirements Coverage and How Can It Be Analyzed?

If you are dealing with the term requirements coverage in the context of systems or software development, you might discover the following ambivalence: on one hand, requirements coverage comes with room for interpretation because it is not well-defined. On the other hand, several well-defined process standards and maturity models demand that coverage has to be measured — especially in safety-critical developments.

In this post I am going to report on some experiences we made when confronted with that ambivalence and on how we approached an “adjustable definition” of the term coverage.

Existing definitions of “requirements coverage”

Let’s look at three different sources for a definition of the term requirements coverage. According to all of them, requirements coverage addresses the direct relationship between requirements and test cases.

A Google search shows several definitions, but they are not universal. Most are related to commercial requirements management tools and thus tool-specific. They do share one thing in common: a requirement is considered covered if at least one test case is assigned to it.

Wikipedia (at least the English and German versions) does not provide any definition of requirements coverage. It does define code coverage as a measure describing “the degree to which source code is executed when a test suite runs.”

Automotive SPICE is a standard to assess the maturity of processes for the development of control modules in the automotive industry. The A-SPICE Process Assessment Model utilises the V-Model and demands that “the selection of test cases shall have sufficient coverage.”

Real-life encounters of requirements coverage

The header image of this article roughly illustrates a development process according to the V-Model.

(Customer test cases are the customer’s responsibility and are therefore not considered.)

In a specific project, the customer demanded regular progress reports based on their customer requirements. They wanted to know how many of them are analysed, implemented, and covered. The terms were defined based on coverage metrics:

• analysed: covered by high-level requirements — a customer requirement must have at least one high-level requirement assigned.
• implemented: deeply covered by low-level requirements — analysed + all derived high-level requirements are covered by low-level requirements.
• covered: deeply covered by test cases — implemented + all derived requirements are covered by test cases.

Let me illustrate this with the following examples:

Customer requirement Cust-Req 3 is covered: high-level requirement HL-Req 11 and low-level requirement LL-Req 18 are derived, both have an adequate test case assigned (HL-Test g and LL-Test d respectively).

Customer requirement Cust-Req 2 is only analysed — it is not considered implemented because HL-Req 19 lacks a low-level requirement.

So what did we learn from this project? The definition of requirements coverage as a “direct relationship between requirements and test cases” is not sufficient:

• Coverage is not only important on the same level of the V, but also along the left leg of the V.
• We have to distinguish between direct coverage (the standard definition) and deep or transient coverage, which requires tracing requirements not only horizontally on the same level (left leg to right leg of the V), but also vertically between levels (from a higher level to a lower level within the left leg).

Our definition of coverage

Summing up, we define coverage as follows:

Based on this definition:

• “Cust-Req 3 is covered by low-level tests.”
• “Cust-Req 2 is covered by high-level requirements.”
• “Cust-Req 2 is not covered by low-level requirements.”

What this definition leaves out

This post is a simplified summary of several project experiences. The following aspects were intentionally left out:

• The V-Model usually has more levels.
• Test results were not considered. Taking results into account, we can additionally analyse whether a requirement is verified.
• Properties of development artifacts matter. For example: a low-level requirement that is not yet “approved” must not be considered during coverage analysis. And sometimes requirements can be flagged as “covered by…” even without links — e.g. by order of the project lead.
• In some projects, requirements derivation is allowed on the same level. This introduces a distinction between split requirements (where the ancestor is covered if all descendants are covered) and derived requirements (where both ancestor and descendants must be linked directly or indirectly to lower levels).
• Even within the same project, different roles lead to homonyms: “high-level requirements are covered” means different things depending on whether the speaker is responsible for high-level tests or low-level requirements.

Our approach: tool-supported traceability

Automotive SPICE states: “Bidirectional traceability supports coverage, consistency and impact analysis.”

If you want — or need — to measure requirements coverage, you first need to establish traceability. But how do you do that efficiently?

Maintaining a traceability matrix is a common starting recommendation. This looks feasible at first glance, but it does not scale. A matrix with n dimensions (one per V-Model level) and thousands of rows and columns quickly becomes unmaintainable.

What you need is a dedicated traceability tool. Good traceability tools:

• recognise artifacts easily so they can be linked readily,
• support users heavily in creating links through efficient search and selection,
• ideally derive links based on similarities,
• extract links already maintained in other tools.

Most importantly: evaluation of trace data must be as easy as possible. Look for a tool that provides:

• Static reporting: ad-hoc and automated reports in multiple formats.
• Dynamic queries: a human-readable query language with expressive features for traversing artifact graphs — usable without knowledge of graph theory.
• Data export: export of raw and pre-aggregated trace data to Excel, databases, or other formats for further analysis.

Requirements coverage analysis in practice

The following screenshot shows an example of a traceability analysis perspective. At the top, a dynamic query; below it, the query result:

The next screenshot shows the same data exported to an Excel sheet, where the data can be further processed and visualised:

This tooling approach makes it possible to efficiently answer the coverage questions outlined above — even in large projects with complex V-Model hierarchies.

Requirements traceability in practice — How end-to-end traceability works in regulated projects, which tools have proven themselves, and when the RTM approach reaches its limits: Requirements Traceability at itemis →