13.1. Principles of program verification and testing

Verification is a process for determining whether software and its components comply with the requirements imposed on them in successive stages of the life cycle of a PS. Analysis, reviews (reviews) and testing from requirements are the most important part of the verification and establishment of program correctness. The main purpose of verification of software is to detect, register and eliminate defects and errors that were made during the sequential development or modification of programs. For cost-effectiveness of resources during its implementation, verification should be integrated as early as possible with the design, development and maintenance processes. It is usually conducted from top to bottom, ranging from general requirements specified in the terms of reference and / or specifications for the entire information system to detailed requirements for software modules and their interaction.

The purpose of the PS verification is to successively verify that in the implemented program package (Fig. 13.1):

• general requirements for the information system, designed for software implementation, correctly processed into the specification of high-level requirements for a set of programs that meet the original system requirements;

• high-level requirements are correctly processed into the PS architecture and in the specification of requirements for low-level functional components that meet high-level requirements;

Requirements Compliance Verification

to technological support of the program life cycle

software requirements

Requirements Compliance Verification

to operational and technological documentation requirements

to software

Fig. 13.1

- specifications of requirements for functional components of the MS, located between the components of high and low level, each time meet the requirements of a higher level;

• the PS architecture and specifications of the requirements for low-level components are correctly revised into the source texts of the program and information modules;

• executable object code satisfies the requirements for the source code of software components.

In addition, requirements for technological support of the life cycle of the substation as well as requirements for the operational and technological documentation are subject to verification for compliance with the specification of requirements for a specific software project.

The objectives of the PS verification are achieved through the sequential execution of a combination of views, analyzes, development of test scenarios and procedures and the subsequent execution of these procedures. Test scenarios are designed to test the internal consistency and completeness of the requirements. The execution of test procedures should provide a demonstration of the compliance of the tested programs with the original requirements. Information on the verification process includes system requirements, requirements for software systems and their architecture, traceability data for sequential requirements conversion, source code of programs, executable object code, software verification plan, software qualification test plan. Verification results should be included in the documents: PS verification procedures performed, description and qualification testing report of the PS and its components.

High-level requirements reviews and analyzes are designed to detect, record and repair defects and errors that were introduced during the sequential development and specification of PS requirements specifications. These reviews and analyzes should confirm the correctness and consistency of high-level requirements, as well as ensure that:

• Fully defined functions of the information system to be performed by the PS;

• requirements for functionality, efficiency and quality of the system are detailed in the initial requirements of the high level of PS and that the derived requirements are correctly identified and their necessity justified;

• each high-level requirement for PS is accurate, unambiguous and sufficiently detailed and that the requirements do not conflict with each other;

• there are no conflicts between high-level requirements and the capabilities of the hardware and software of the object calculator, especially such as the system response time and the characteristics of the input / output equipment;

• the process of developing requirements for PS is fully consistent with the standards for creating specifications of requirements and any deviations from the standards are justified;

• Functional and structural characteristics of quality, intended for software implementation, are fully included in the high-level requirements for PS.

The views and analyzes of the source texts of programs are designed to identify and record defects and errors that are made during programming of the PS components. These procedures should confirm that the output of the coding algorithms - the texts of the programs are accurate, complete and can be checked. First of all, the correctness of the program text with respect to the initial requirements and the PS architecture and compliance with the programming standards should be determined. These views and analyzes are usually limited to the source code of the programs, which confirm that it is correct, complete and meets the low-level requirements for the components, as well as the fact that the source text does not contain unnecessary and undescribed functions. It should be verified that the program development process was fully implemented in accordance with the programming standards and deviations from these standards are justified, especially in cases of limited complexity and use of program designs that must meet the goals of correctness of the system. In this context, complexity is understood as the degree of coherence of software components, the level of nesting of control structures, and the complexity of logical and numerical expressions. This analysis should ensure that possible deviations from the standards are justified.

Testing PS from requirements has two complementary goals. The first goal is to show that the PS meets the specified requirements for it. The second goal is to show with a high degree of confidence that

defects and errors that could lead to unacceptable failure situations affecting the correctness and reliability of the system were eliminated. Testing the integration of software components, based on the requirements, should focus on the relationship between the requirements for the software and the implementation of the requirements for its architecture. The purpose of such testing is to ensure that the software components interact with each other correctly and satisfy the requirements for the PS and its architecture.

Years of experience have shown that it is not possible to create a single universal testing method and an ordered series of significantly different methods should be applied. Each method has different test objectives, verifiable program components, and results evaluation methods. Only joint and systematic application of various testing methods allows us to achieve high quality of functioning of complex program complexes.

When choosing and applying methods of testing software components, it is necessary to take into account the general requirements for them and their features:

• the relatively high proportion of the creative work of testing specialists leads to the need to ensure their highly effective interactive interaction with the means of automation of checks;

• the unpredictability of the types and places of detected errors in the programs limits the possibility of their automatic detection and determines the need to focus on partially automated methods and tools with a creative, high role of specialists in testing;

• a variety of possible locations and types of errors, with a relatively rare detection, leads to the registration and analysis of a large amount of redundant information about the process of program execution during testing;

• the high complexity of the debugged software components, the creative and iterative nature of the testing process make it difficult and limit the possible accuracy of the assessment of the completeness of the testing and the quality of components achieved;

• active participation in testing specialists differing in qualifications, experience, temperament and creative possibilities, as well as differences in architecture, complexity and functions of

gram components do not allow to strictly regulate the complexity, methods and technologies for the use of types and test automation tools.

In the process of developing programs, specialists should strive to cover testing the functioning of software systems in the whole available area of ​​change of initial data and application modes. At the same time, the nomenclature and ranges of variation of tests are limited either by the permissible duration of preparation and execution of tests, or by the computing resources available for use for the purpose of monitoring and testing in normal operation. This is reflected in the choice of test methods and the sequence of analysis of the components of the PS, the volume of applied tests and the depth of testing. The choice of effective testing methods and the sequence of their application are most influenced by the main characteristics of the tested objects:

• the class of the program complex, which is determined by the depth of the connection of its operation with real time and random influences from the external environment, as well as the requirements for the quality of information processing and reliability of operation;

• the complexity or scale (dimensions) of the program complex and its functional components, which are the final results of the development;

• The predominant elements in the programs are those that compute complex expressions and transform measured values ​​or process logical and symbolic data to prepare and display solutions.

Test options should be designed to verify the correctness of operation and determine the conditions under which potential errors may occur. Analysis of test coverage requirements for the PS should determine which requirements have not been tested and which software structures were not executed during testing. The test coverage analysis consists of two steps, including the analysis of the coverage based on the requirements and the analysis of the structural coating. The first step analyzes the test suites against the requirements of the PS, to confirm that the selected test suites satisfy the established criteria. This requirements based coverage analysis should determine how complete the test

Qing checked the implementation of all requirements in the specification for the PS, and show the need for additional test suites. Test options based on requirements may not fully cover the structure of the program. Therefore, an additional analysis of the structural coating is performed and its verification is carried out. The second step is to confirm that the testing procedures based on the requirements covered the entire program structure. Structural Coverage Analysis should determine whether elements of the program structure that are not verified by test procedures based on requirements are missing.

Further attention is focused on the main methods of quality assurance regarding simple programs. These include individual software modules (PMs) or their small groups that solve a fairly simple functional task. However, these components do not differ below, and the term “software module” is mainly used. These test objects have a number of fundamental features that distinguish them from complex software tools and directly affect the methods used and test automation tools.

The low dimensionality of the PM provides their visibility and the possibility of a detailed analysis of the functions, program structure and task solving process. Detailed control of the PM can be carried out systematically and deterministically with the accuracy of any operator or operand in the program. The desire to reduce the complexity of testing is implemented partly by consolidating the observed and analyzed components to the level of small groups of operators in a programming language within one structural element of the program. Thus, the control has access to all the logic of solving the problem and all possible routes and variants of program execution. As a result, when testing, it is possible to assess and control the degree of testing carried out and the achieved quality of the components of the PS, as well as to highlight the following types of testing.

Testing for error detection is aimed at identifying deviations of the results of the functioning of a real program from the specified requirements and reference values. The task is to detect the maximum number of program defects, which take any deviation of the results from the standards. Successful is

tirovanie, which leads to the detection of the existence of errors at minimal cost.

Testing to diagnose and localize errors is intended to accurately establish the original location of the distortion of programs or data, which is the cause and source of the deviation of results from standards when testing for error detection. Effective are the tests that contribute to the rapid and accurate localization of the primary defects of programs and data. At this stage, the costs are justified and testing can be considered successful if it led to the definition of program elements or data that are subject to immediate adjustment.

Methods and strategies for testing software components are used to process the text of programs in various forms and to present information about test results in a form suitable for analysis by specialists. The form of presentation and execution of the program for testing depends on the stages of development, the level of programming languages, the availability of appropriate automation tools and other factors. Programs in the testing process are used in two fundamentally different forms of presentation:

• in the form of text in a programming language or a formalized description of requirements specifications (symbolic representation), convenient for human analysis and usually inaccessible for direct receipt of the results of functioning on a computer;

• in the machine code of a specific computer (object representation), suitable for automatic processing of certain code source data and inconvenient for human analysis.

Primary and secondary errors are detected at successive stages of testing (see Lecture 10). Secondary errors are decisive for the quality of the functioning of the programs, since not every primary error makes a significant contribution to the distortion of the output results. As a result, a number of primary errors may remain undetected and, in essence, do not adversely affect the functional characteristics of the programs. An essential feature of testing PM is the relative closeness of the manifestation of secondary errors to their causes — the primary errors. This means that the consequences of errors are found in the distortion of the results directly in place.

localization of the primary error or after a small number of program execution steps. This greatly facilitates their diagnosis and localization, as well as monitoring the correctness of the adjustments made.

Program analysis at the level of symbols and programming language operators provides all types of testing: for detecting errors, for their localization and for checking the correctness of the corrections made. This feature is used not only for autonomous debugging of modules, but also for complex testing and detection of PS functioning defects. In the latter case, one has to iteratively drill into the checked components, up to the module and its operators. At the same time, in order to localize and correct the primary error, in some cases it is necessary to return to testing modules at the level of program operators.

Another feature of testing PM is the need and the possibility of providing guarantees of their high quality and suitability for use in various PS projects. Prospects for repeated use of proven software components in various versions of software systems can only be ensured with a clear formalization of their functions and conditions of use, as well as with confidence in the correct implementation of functions and interfaces in a particular environment. As a result, systematic testing and formalized control of the achieved correctness after each test cycle is necessary. Complete the debugging and testing of the PM should be their certification, with the application list of implemented requirements, interfaces, characteristics of completeness of testing and ranges of variation of test data. Such certification can serve as a guarantee of quality for repeated use of the PM in various hardware and operating environments.

Testing of control flows (program structure) should be the initial stage, since with an incorrect structure, the most gross distortions of the output results and even the absence of some of them are possible. It consists in checking the correctness of the sequences of control transfers and the formation of program execution routes when processing tests. Testing the structure of programs in most cases requires relatively lower costs compared to testing on data streams.

Testing data streams can be divided into two stages. The first stage of testing consists in analyzing the data processing that determine the values ​​of predicates in logical decision-making operators. These decisions affect the information processing routes, which brings together in this part the method of testing data streams with testing the structure of the program. The second stage of testing data processing consists in checking calculations using analytical formulas or numerical values ​​of the results depending on the numerical or logical values ​​of the initial data. The results of manual or automated calculations using the same or similar formulas are used as standards.

Any methods of testing PM, to a greater or lesser extent, are focused on the detection of errors of certain types. Methods of testing control flows are intended primarily for detecting errors in the structure of the PM and the implemented information processing routes. Methods of testing data streams provide detection of errors in the computational part of programs and in the processes of converting various information. These testing methods are used both when presenting programs in programming languages ​​and when they are executed in object (machine) code after translation. Such an orientation makes it possible to rationalize the sequence of application of methods in order to eliminate, first of all, errors that most affect the correctness of programs, and also focus on methods that allow solving particular problems of achieving the required quality at minimum costs.

A set of test specifications can be considered as the second , independent description of the content and implementation of the successive procedures of the program package. The life cycle and development of tests in the development and maintenance of PS should take place in time, parallel to the dynamics of change and the life cycle of program texts. Tests and scenarios for their implementation should be adequate and fully reflect the content of the texts of the components of the program complexes, but in a different form. They should be considered and analyzed as another form of describing the functions of programs and PS data, which may also contain defects and errors. Thus, the program (procedural) presentation of the PS should fully comply with its

equal content in the form of scripts and tests to verify their mutual compliance. In this case, defects and errors are possible in both forms of describing the content of programs, determining their location and elimination is the main task of testing.

Specification of requirements for the functions and quality of the system

Conformity verification

requirements specifications

software requirements

to the functions and quality of the system

Verification of completeness

and correctness of the specification

and system test scripts

Conformity verification

requirements specifications

to functional components

software requirements

Verification of completeness

and correctness of specifications

and test scripts

software tool

Verification of completeness

and correctness of specifications

and test scripts

software components

Conformity verification

requirements specifications

to program and information

modules requirements

to functional components

Verification of completeness

and correctness of specifications

and test scripts

software

and information modules

Object compliance verification

code specification requirements

to software texts

and information modules

Fig. 13.2

In order to ensure high quality of the software product, in parallel with the verification of requirements and the programming of adjustments, it is advisable to develop and verify specifications and test scenarios reflecting the methods and specific procedures for verifying the implementation of changes to these requirements (Fig. 13.2). Test specifications can be used to check consistency, internal consistency and completeness of the implementation of requirements without executing programs. For each requirement for the software package, its architecture, functional components and modules, a test specification should be developed to ensure validation, adequacy and the ability to subsequently test the component for compliance with this requirement. This mutual verification of the functions of the components, as reflected in the requirements and specifications of the tests, provides for an increase in their quality, reduction of defects, errors, ambiguities and contradictions.

When testing programs, it often turns out that not every requirement in the specification is described quite fully and correctly so that it can be verified by tests. When developing tests based on such specifications, due to their possible uncertainty, it may be found that not every test requirement for a program or data can be prepared. On the other hand, not every test may have an adequate requirement for the functions of the PS in the specification. Test specifications should provide additional control over the correctness of requirements specifications and verification of the interaction of components at an appropriate level of PS description. Independent development of test specifications based on requirements specifications creates the basis for detecting which requirements have not been tested or cannot be verified by testing in principle. Thus, verification of requirements specifications for software components and software can be used for two purposes (see Fig. 13.2):

• for development, programming and verification of program texts and interfaces for interaction of program components of different levels in a program complex;

• to create a coordinated set of tests for a set of components that provide mutual verification of the implementation of specifications of requirements for a set of programs and components.

As a result, a set of specifications for test requirements can be used in development and maintenance as benchmarks and the second adequate form for describing program content for end-to-end verification of test requirements specifications from top to bottom, as well as themselves being verified for correctness of compliance with the original requirements for program text components and data of different levels . Such parallel mutual checks of requirements specifications and program texts and test specifications contribute to the identification and elimination of many secondary defects and errors in software systems. Subsequently, these test specifications should be used to directly test compliance with the requirements for software components of the appropriate level. In addition, parallel and independent development, on the one hand, of program specifications and test specifications, as well as their implementation, on the other hand, allows parallelizing the work of the life cycle of the substation, which leads to a reduction in the development time of components and program complexes.

The implementation of these objectives of the interaction of verification and testing can be done by different methods and independent experts - programmers and testers, which allows you to use the results of their activities to compare the content of the same programs presented in programming languages ​​and described in test languages. The features of program descriptions and implementations, as well as programmers' thinking, are based on the functions and procedures of program execution that differ significantly from the ideas and methods for describing the same program functions by testers who created test scripts. They focus on specific procedures for checking the functioning, possible results and interaction of PS components. This allows you to identify secondary defects that appear during adjustments, and improve the quality of development and maintenance by comparing the two methods and results of the description of the same programs, due to the fact that the probability of identical errors in test scenarios and in the implementation of program texts is small.