Controlling the Functional Testing of an Operating System

Abstract

Functional testing of operating systems is in transition from a predominantly imprecise art to an increasingly precise science. The process that controls this testing is maturing correspondingly. The laissez-faire approach is giving way to a disciplined approach characterized by rigorous definition of the test plan, systematic control of the test effort, and objective quantitative measurement of the test coverage. This paper describes just such a disciplined test control process, which is composed of five steps: 1) the survey, which establishes the intended extent of testing; 2) the identification, which creates a list of functional variations eligible for testing; 3) the appraisal, which ranks and subsets the eligible variations so that test resources can be directed at those with the higher payoff; 4) the review, which calculates the test coverage of the test case library; and 5) the monitor, which verifies attainment of the planned test coverage. Throughout the test process, specification testing is distinguished from program testing.