Cleanroom software development

Title: -. Author: -. Subject: -. Keywords: -. Creation Date: -. Modification Date: -. Creator: -. Under incremental development, customers provide feedback on the evolving system by direct operation of user-executable increments. The relative clarity of requirements may influence an increment plan in two ways.

Volatile requirements may be implemented in an early increment, so they can be clarified. If the user interface is not well-established, for example, it is an ideal candidate for an early increment. Alternatively, unstable requirements may be planned for later implementation, when questions affecting the requirements have been settled. Requirements to be settled by concurrent research e. A functional usage distribution is developed as part of a top-level Cleanroom specification.

Expected usage probabilities of system functions are established from historical data and best estimates provided by customers. System functions with high expected usage probabilities will receive greatest exposure in the field, and may therefore benefit from the greatest exposure to testing. Since increments are cumulative, the functions developed in early increments will be tested several times i. System functions expected to receive the greatest operational usage by customers, therefore, are candidates for early increments.

Increasingly, customers are specifying formal software reliability requirements. Reliability "sensitivities" and allocations can be calculated for subsystems, and subsystems that will have the greatest impact on total system reliability may be candidates for an early increment. The minimal machine is built in the first iteration, and is enhanced in subsequent iterations until the complete machine has been built.

Incremental development of software is entirely compatible with this standard approach to hardware development. A machine must be powered-on, for example, before it can be used.

The software for system start-up, therefore, would likely be among the functions implemented in the first increment of an embedded software project. In most applications there is some logical allocation of functions to increments based on relationships among functions.

In a database application, for example, an add must precede a delete. In a statistical application, data must be entered or retrieved before it can be analyzed. Although program stubs i. Novel or particularly complex work may pose a risk to the schedule or even the viability of a project. If such work is scheduled for an early increment, experience will either lend support to existing plans or point to the need to revise plans.

If aspects of the project are not novel or complex in absolute terms, but are novel or complex relative to the experience of the team, an early gauge on feasibility is still desirable.

The Cleanroom process emphasizes economy of effort through use of "common services" certified reusable components across and within systems. When existing components are identified as potentially reusable, the development team must evaluate the relative effort required to tailor the component for use in the new system vs. If the evaluation is in favor of the existing component, the team may want to use the component in an early increment in order to validate its expected performance.

New common services may be desirable candidates for an early increment as well. Since common services may be in multiple places in a system, they have a disproportionate impact on system reliability relative to other, single-instance components. Since objects may be reusable parts, the rationale for object development in an incremental development plan follows the rationale for reusable components in general.

Risk analysis is used to determine the size and content of each increment. Following a risk assessment, the project team defines an increment plan of one or more increments. Each increment mitigates some risk of project failure. For example, if the user-interface requirements are poorly understood, then the project may be at high risk of delivering a hard-to-use product.

To mitigate this risk, the early increments would contain mostly user-interface code, but not much "function," and could be shown to selected customers. Different increment plans are used when requirements stability or performance properties are high risks. Incremental development has many advantages over bottom-up and traditional "waterfall" life cycles:.

The fundamental approach to verification as espoused by Cleanroom is aimed at introducing mathematical reasoning, not mathematical notation into the verification process. The principal motivation is to provide a rigorous methodology for software development and to provide a firm foundation as an engineering discipline. Mathematical verification of programs is done by using a few basic control structures and defining proofs following rules specified in a correctness theorem.

The proof strategy is divided into small parts that easily accumulate into proof for a large software system. The method of human mathematical verification used in Cleanroom is called functional verification. Functional verification is organized around correctness proofs, which are defined for the design constructs used in a software design.

Using this type of functional verification, the verification problem changes from one with an infinite number of combinations to consider to a finite process because the correctness theorem defines the required number of conditions that must be verified for each design construct used. It reduces software verification to ordinary mathematical reasoning about sets and functions.

The objective is to develop designs in concert with associated correctness proofs. Designs are created with the objective of being easy to verify. A rule of thumb followed is that when designs become difficult to verify they should be redone for simplicity. Statistical quality control is used when you have too many items to test all of them exhaustively. Instead, you statistically sample and analyze some items and scientifically assess the quality of all of the items through extrapolation.

This technique is widely used in manufacturing in which items in a production line are sampled, the quality is measured, then sample quality is extrapolated to the entire production line, and flaws are corrected if the quality is not as expected. For software, this notion has been evolved so that you perform statistical usage testing--testing the software the way the users intend to use it.

This is accomplished by defining usage probability distributions that identify usage patterns and scenarios with their probability of occurrence. Tests are derived that are generated based on the usage probability distributions. System reliability is predicted based on analysis of the test results using a formal reliability model, such as mean-time-to-failure. The underlying concern is that random, statistical-based testing will not provide sufficient coverage to ensure a reliable product is delivered to the customer.

The coverage concern stems from a misapprehension that statistical implies haphazard, large, and costly and that critical software requirements, which may be statistically insignificant, are overlooked or untested. Coverage is directly related to the robustness of the usage probability distributions that control the selection process and has not proven to be a problem in current applications of the methods.

In one study performed on the level of requirements coverage using statistical testing, percent of the high- level requirements were covered, 90 percent of the subcomponent-level requirements were covered, and approximately 80 percent of all requirements were covered. The Cleanroom method asserts that statistical usage testing is many times more efficient than traditional coverage testing in improving the reliability of software. Statistical testing, which tends to find errors in the same order as their seriousness from a user's point of view , will uncover failures several times more effectively than by randomly finding errors without regard to their seriousness.

The basis for software reliability starts with the definition of a statistical model, generally based on the concept that input data comes in at random times and with random contents. With defined initial conditions, any such fixed use is distinguishable from any other use. These uses can be assembled into a sequence of uses, and the collection identified as a stochastic process subject to evaluation using statistical methods.

Coverage testing is anecdotal and can only provide confidence about the specific paths tested. No assessment can be made about the paths not tested. Because usage testing exercises the software the way the users intend to use it, high-frequency errors tend to be found early. For this reason, statistical usage testing is more effective at improving software reliability than is coverage testing. Coverage testing is as likely to find a rare execution failure as it is to find a frequent one.

If the goal of a testing program is to maximize the expected mean-time-to-failure, hence the reliability of the system, a strategy that concentrates on failures that occur more frequently is more effective than one that has an equal probability of finding high- and low- frequency failures.

Experimental data from projects where both Cleanroom verification and more traditional debugging techniques were used show that the Cleanroom-verified software exhibited fewer errors injected.

Those errors were less severe possibly attributable to the philosophy of design simplification and required less time to fix.

Most software users do not care how many defects are in the code. They care instead about how often the software fails to meet their needs, how severe each failure is, and how long the repairs take. Cleanroom testing adapts statistical quality control techniques to measure these quality characteristics, enabling targeted quality improvements. In precision manufacturing, statistical quality control begins with a precise specification. A statistical sample of the manufactured product is measured against the specification, and the quality characteristics of that sample are used to estimate the overall quality of the manufacturing run.

The quality of a software product does not depend on variations in the physical copies of the product, but in its execution behavior. In Cleanroom, testers statistically sample these behaviors by providing appropriate inputs, and measure each against the specification. Data from the testing process is entered into a reliability model that predicts the quality of the software in the field.

The challenge of statistical testing lies in sampling the executions. Want more? Advanced embedding details, examples, and help! The 'cleanroom' software development process is a technical and organizational approach to developing software with certifiable reliability. Key ideas behind the process are well structured software specifications, randomized testing methods and the introduction of statistical controls; but the main point is to deny entry for defects during the development of software.

This latter point suggests the use of the term 'cleanroom' in analogy to the defect prevention controls used in the manufacturing of high technology hardware. Box structure specification The specification method uses box structure. Box structure is used to describe the functional specification. The box structure separate and isolate the behaviour, data and procedure in each increment.

Formal design The cleanroom design is a natural specification by using the black box structure approach. The specification is called as state boxes and the component level diagram called as the clear boxes.

Correctness verification The cleanroom conducts the exact correctness verification activities on the design and then the code. Verification starts with the highest level testing box structure and then moves toward the design detail and code.

The first level of correctness takes place by applying a set of 'correcting questions'. More mathematical or formal methods are used for verification if correctness does not signify that the specification is correct.