Summary
This is a discussion of how to ensure a requirement is testable. There is also a description of Test Protocol documents
Testability
A Requirement is considered Testable if all the following are met:
- There is an appropriate list of set up instructions that can make the Condition(s) true.
- If there are Compound Conditions, then the setup instructions can make all the Condition(s) true at once.
- When the Condition is true, the Assert can be determined to be either true or false
- The Assert is either true or false, and cannot be both true and false at the same time.
1) All Conditions Can be True
It must be possible, somehow, to make the Condition or Conditions all true. If not, the requirement can't ever be tested and therefore is invalid. Invalid requirements must be removed from the requirements document.
Here are some (contrived) examples:
REQ-100 On the Foo Screen, the label1 shall display "bar"
!! The Foo Screen does not exist in this application
REQ-101 If the value of field1 is 0 and the value of field1 is 21, the label2 shall be "hello"
!! It is not possible for the Compound Condition to ever be true
REQ-102 If the sensor value is greater than 200, label3 shall be "error"
!! The sensor in use can not measure values above 199.
REQ-103 If the Bar function is enabled, hide the label1 field
REQ-104 On the Foo Screen if the Bar function is enabled, the label1 shall display "bar"
!! The Foo Screen exists, the Bar function is enabled, but the label1 field can not be seen at that time
There may need to be a reasonableness standard applied to the requirement as well. It may be "possible" to make the conditions true, but that scenario is extremely rare, or it is short-lived. Or there may be a set of instructions that make the conditions true, but they take the tester a very long time to run, the requirement may still be untestable from a practical point of view.
2) Assert must be determinable
There must be a method to determine if the Assert is true or false. Here is a (contrived) example.
REQ-200 When the hard drive is powered off, the hard drive shall contain the file "some_file.txt"
!! Can't check the contents of the hard drive unless it is powered on
In this example, since the hard drive can't be checked while it is powered off, the existence of the file can't be determined. The Requirement is therefore untestable.
Note there may be extreme methods available for a tester to use. For example, the hard-drive platter could be tested independently using a secondary sensor and therefore the file existence could, in fact, be determined. And so, again, there may need to be a reasonableness standard applied to the requirement.
3) Assert must be either true or false, not both
The tester must be able to make a clear determination the system is meeting the Assert or not.
Some (contrived again) examples:
REQ-300 The Startup screen shall display for longer than 2 seconds and less than 1 second
!! The assert is mutually inconsistent
REQ-301 The Next button shall be blue.
!! The Next button button is in fact blue, but there are several kinds of blue buttons in the system.
A Tester may need specific expertise to determine if the Assert is being met. That expertise is usually gained through training. In this case, there may be a prerequisite in the Test Case indicating that the Tester has finished the training and provides evidence of the same in the Test Result.
A Test Case may require multiple runs to be done and a statistical method applied to determine if the Assert is being met, e.g.
Req-400 The therapy shall deliver 1L of fluid in 120 seconds +/-5 seconds at most and +/- 3 seconds at least 70% of deliveries.
The Test Case would run the test, say 10 times, and measure the time it takes to deliver 1L. For all deliveries, the time to deliver must be 115 - 125 seconds, and at least 7 times it would be 117 to 123 seconds to pass the test case.
Test Protocols
A Test Protocol is a set of instructions for testing the requirements.
A Test Protocol is usually divided into a set of Test Cases. Each Test Case performs tests for a small number of requirements (usually one or two).
The Test Case instructions will prepare the system such that the Conditions for the requirement are all true. Once the Conditions are true, the Assert is evaluated to produce a Test Result. An Assert produces one and only one Test Result.
If there are a sufficient number of Test Cases with common setup instructions or from an organizational point of view (e.g. "test the Basic Therapy"), they may be grouped together into a Test Suite. A Test Suite's result is an aggregate of its individual Test Cases e.g. "Basic Therapy is Passing".
Test Result
A Test Result has two values: Pass or Fail.
- If the Assert is true, then the Test Result is Pass
- if the Assert is false, then the Test Result is Fail
Note for logistical convenience, a Test Result may have other values. For example:
- TODO: indicates there is more work to be done, e.g. used during test development
- DEVIATION : indicates this is a known failure, e.g. the system will ship with the deviation
- SKIP: indicates this is a temporary skip of the Assert, e.g. the Assert is known to fail and will be fixed at some time later
Using Requirements
In a Waterfall software methodology, Requirements are tested when they are complete, when the design is done and the implementation is complete. There are specific times in the development flow where a review is done of the Requirements, Test Protocols, etc.
In an Agile methodology this is done in an ongoing process. For example, to implement a Feature:
- Prescriptive Requirements are reviewed to ensure the feature is outlined correctly there. If not, additional requirements are added, modified or deleted
- an informal design session is held to determine how the Feature will behave and how it fits in to the existing system.
- Existing Descriptive Requirements are reviewed to determine if any need to change because of the new Feature.
- New requirements are added for the Feature
- The developers, testers, user representatives and other interested parties review the changes to ensure they are correct.
- Design documents are similarly updated and development is started. If any questions arise, the requirements are reviewed for updates as necessary.
- Test Protocol and Test Cases are modified or updated to test to the new Requirements. If any questions arise, the requirements are reviewed for updates as necessary.
- The development team delivers an initial version of the software to the test team.
- The test team runs their initial version of the Test Cases.
- This is repeated until all the Tests pass and the following need no further changes:
- Requirements
- Test Protocols
- Test Cases
- the software