14.1. Processes evaluations characteristics and software tests funds

For evaluating the characteristics and testing of software at various stages of the life cycle, as a methodological basis, it is advisable to use the recommendations of ISO 14598: 1-6 - Evaluation of a software product. The PS assessment process in the standard is presented as a set of actions performed in cooperation by the customer and the evaluator-tester. Potential customers of software evaluation can be developers, suppliers, customers, users of software systems, manufacturers of information processing systems, as well as software testing arbitration laboratories. To obtain the greatest effect from the test results, it is recommended that the assessment be as possible:

• objective - the results of the assessment should be based on real facts that are not tinged with the feelings or opinions of the testers;

• repeatable - re-evaluation of the identical product for the identical specification by the same tester should give the same results as in the initial assessment;

• reproducible - evaluating the same product for the same specification by different specialists should give the same results as in the previous test;

415

Lecture 14. Integration, qualification testing and testing of complexes ...

- impartial - the performers of the assessment process should be open-minded to any particular result.

The general scheme of the processes of evaluating the characteristics of program complexes is (Fig. 14.1):

Formalization of initial requirements for software evaluation :

• determination of the objectives of testing and quality assessment at the intermediate and final stages of the life cycle of the PS;

• identification of the type of software;

• identification of users of assessment results;

• highlighting the features of the assessment model and the composition of the required quality characteristics

I

Formalization of principles for evaluating the characteristics and testing of software:

• selection of software characteristics and typing requirements for measurement and testing;

• the establishment of priority levels of characteristics and quality attributes;

• selection of criteria for examination and comparison of quality characteristics with requirements

*

Designing the process of evaluating and testing software features:

• developing plans for assessing the characteristics of the software in accordance with the needs of users and the stages of the life cycle;

• design of software performance evaluation and testing processes

l

The implementation of processes and the use of results of evaluating the characteristics of the software:

• performing tests to assess the real values ​​of the characteristics of the software;

• comparison of test results with the criteria for the required characteristics of the software;

• evaluation and generalization of results of tests of software characteristics

Fig. 14.1 416

14.1. Software performance evaluation and software testing

• formalization of the initial requirements for the evaluation of the values ​​of the characteristics of the software, the definition of test objectives, the identification of consumers of test results;

• formalization of the principles and features of evaluation during the examination, measurement and testing of the characteristics of the software, highlighting the criteria for comparing the obtained characteristics with the requirements;

• planning and designing the processes of evaluating the characteristics in the life cycle of a software tool in accordance with the needs of the users of these characteristics;

• implementation of the processes of testing, measuring and evaluating the achieved quality of a software product, comparing the test results with the requirements; design and use of results.

The first part of the standard introduces the concept of planning and managing the processes of evaluating program characteristics, as well as their relationship with the PS life cycle management processes (according to ISO 12207). In preparation for testing, it is recommended to structure the technology and procedures for applying a specific PS in order to consistently evaluate in detail the functional characteristics groups or individual quality attributes during the life cycle stages.

The decision on the evaluation of the characteristics of the PS can be made in the development process and the entire life cycle. If such a decision is made at the initial stage of development, then it becomes possible to embed tests and measurement tools for testing in the development process. This ensures maximum success in meeting all the requirements regarding the results of evaluating the characteristics of the PS and minimizes the risk of errors in extreme, unplanned situations. When a PS developer is a customer, an early contact with an appraiser for testing or examination can help the developer to take into account some special requirements from the tester. For very large, complex PS projects, it should be beneficial for the developer to have detailed cooperation with the performance evaluator during the entire product development process to minimize the length and cost of the testing process. Customer's responsibility for testing should be:

- establish its necessary legal rights to perform PS tests;

417

Lecture 14. Integration, qualification testing and testing of complexes ...

• provide the tester with the information necessary to identify and describe the software and components;

• establish initial requirements and enter into negotiations with the tester to develop real evaluation requirements;

• requirements for testing should be subject to the relevant regulatory documents and standards;

• establish confidentiality requirements for information submitted for testing;

• if necessary, provide support to the tester, including training and access to relevant documents;

• to guarantee the timely delivery to the tester of the description and components of the PS, including documentation and other materials;

• inform the tester of any possible factors that may devalue the test results.

The tester should review the technical constraints related to the measurements or verifications specified in the estimation specification and document the appropriate method. The assessment method is a procedure that describes an action performed by a tester in obtaining the result of a specified measurement, examination, or verification for a system, component, or software. When an estimation method is described on the basis of software tools, this tool should be identified in the test plan and test method. In this case, the duties of the evaluator - tester include:

• check the legal rights of the customer for the system and the PS for evaluation, for which the tester may require the relevant documents from the customer;

• keep confidentiality of all information transmitted by the customer, including the texts of the PS, the records of the results and the test report;

• provide qualified and trained personnel to perform the tests;

• provide tools and technology assessment;

• perform tests in accordance with the evaluation requirements and customer specifications;

• keep records of any work performed during the assessment that affect the results;

418

14.1. Software performance evaluation and software testing

- to ensure the visibility of the tests for observation by the customer and the timely transmission of the evaluation report to the customer.

Evaluation requirements should contain a general description of the scope of the PS and consist of a list of requirements for the characteristics. The relative importance (priority) of specific characteristics in the requirements should be indicated. For each provision, the evaluation requirements must contain a specification of the information contained in the components of the PS.

The evaluation - test specification should define the scope of the examination and measurement of the various components of the product being transferred for evaluation. The level of detail in the test specification should be such that, on its basis, repeatability and reproducibility of tests are guaranteed. The customer must submit a description of the product being evaluated. The purpose of this description is to determine the scope of the assessment and the identification of those components that are considered as part of the product, as opposed to PS components, which are only referred to in order to facilitate understanding of the functions of the product. This should allow to decompose the estimated requirements into subcharacteristics. The tester must then specify the measurement and examination methods for analyzing the characteristics, sub-characteristics and quality attributes of the selected components. The evaluator shall verify the assessment specification in accordance with the requirements, establish that the components listed in the product description contain all the necessary information to perform the assessment in accordance with the requirements.

Test plan or examination of the characteristics of software products should consist of sections:

• introduction - statement of the problem and the purpose of evaluation - testing;

• methodology for ensuring the objectivity of testing the characteristics of the PS;

• the characteristics and attributes of quality and safety of PS for the project according to ISO 9126: 1-4 and other standards;

• Required quality of PS and reliability of testing processes;

• schedule of works on testing the characteristics of PS;

• distribution of duties and responsibilities of specialists in assessing the characteristics and attributes of quality;

419

Lecture 14. Integration, qualification testing and testing of complexes ...

• analysis and use of test results;

• content and execution of test reports;

• additional requirements for equipment, tools and test service methods, for compliance with standards and organizational support for measurements and examinations.

The tester should provide input for the assessment process: predefined assessment specifications; methods and tools. The assessment process - tests are recommended in the standard of the following:

• analysis of the assessment requirements, in which real customer requirements for testing are allocated for identification;

• specification of processes and possible outcomes in which an assessment specification is developed based on the requirements and description of the software product provided by the customer;

• the design of test processes that develops a plan based on the assessment specification, PS components and methods proposed by the tester;

• the process of implementation of the assessment plan, which consists of modeling, examination, measurement and testing of the components of the PS in accordance with the plan, using software tools, as well as actions carried out by the tester, which are recorded and the results recorded in the report;

• conclusion of the assessment, which is to provide a report on the results of tests of components and PS.

The testing process may require the use of software tools to collect the processed data or to interpret intermediate data. Staff appraisers should be trained to use the appropriate tools. When evaluating , the following data should be received, recorded and presented to the customer :

• requirements that describe the purpose of the assessment, in particular, the required criticality and safety of the tested product;

• an assessment specification, which defines the entire analysis and measurements performed, and all the components of the PS that are to be analyzed and measured;

420

14.1. Software performance evaluation and software testing

• assessment plan, which describes the operational procedures necessary to perform the evaluation specification, in particular, all the methods and tools used in the evaluation;

• evaluation records, which consist of an evaluation plan and detailed actions of the evaluator during the implementation of the plan;

• report on the evaluation of the characteristics, which contains the requirements, the specification of the assessment, the results of measurements and analysis, as well as any other information necessary to be able to repeat or reproduce the assessment, as well as a summary report to the customer.

The purpose of the conclusion is to review the test report and transfer the evaluation data to the customer. A joint review and analysis of the report by the customer and the tester should be organized. The customer should be given the opportunity to make comments on the report. If such comments are made, they should be entered into a special section of the report, after which it should be transferred to the tester and the customer.

From the standpoint of different consumers of the results of measuring and evaluating the quality of PS, the third, fourth and fifth parts of ISO 14598 are built: 1-6 - respectively for:

• developers - evaluation of internal and external characteristics of quality (part 3);

• operational users - measurement of external metrics and metrics in use (part 4);

• customers and testers - definition of metrics in use (part 5).

In each part, such sections are highlighted and detailed: the features and needs of specific users in the test results and the nomenclature of the required PS characteristics; concept of testing and quality assessment; determination of requirements for the testing of program characteristics; identification of the characteristics and quality attributes of PS for specific users of test results.

The results of the evaluation of characteristics are proposed to reflect from the perspective of: life cycle processes; products and their components; functioning and application of PS. Requirements for assessment processes are recommended to be structured into main (functional), organizational, project, as well as highlight internal and external quality metrics and

421

Lecture 14. Integration, qualification testing and testing of complexes ...

their measurement, focusing on subcharacteristics and their attributes in the relevant parts of the ISO 9126 standard : 1-4. The implementation of software testing processes according to the requirements of the standard should be carried out by qualified and certified specialists, independent of the project developers , PS creation processes and its components, but correlated with the stages of the life cycle of a specific project in accordance with the applicable ISO 12207 version .

Attention is paid to the expediency of taking into account and using the history of development and changes in the operational requirements of customers-consumers to a specific project and functional purpose of the substation. In the test report, it is recommended to present the quality characteristics in accordance with the nomenclature and measures of subcharacteristics and attributes adapted to the purpose, functions and features of a specific PS project and consumers of measurement results.

The confidentiality of all components and PS test documents shall be protected by an evaluator. Confidentiality requirements cover many aspects of valuation work: including receiving, managing, storing, and transmitting all information regarding product characteristics. The confidentiality of the intermediate test data should be protected, as well as the confidentiality of the originals of the components and documents. The tester should make an effort to prevent any accidental, erroneous or malicious modification of this data.

14.2. Organization and methods for evaluating the characteristics of complex program complexes

The characteristics of the quality of the functioning of software depend not only on their internal properties, but also on the properties of the external environment in which they are applied (see ISO 12119). To reduce uncertainties and direct errors in evaluating the quality of the PS, it is necessary prior to testing to determine the main parameters of the external environment for which the program complex should work with the required characteristics when evaluating its quality and operation.

422

14.2. Organization and methods for evaluating the characteristics of complex program complexes

For this, the customer and the developer should jointly structure, describe and coordinate the environment model and its parameters on average, typical PS application mode, as well as in the most probable and critical modes, in which the required PS performance characteristics should be provided. Such a model should reflect and capture characteristics:

• external flows of information, including their distribution by types of sources, characteristics of data quality and the possibility of their defects;

• the intensity and structure of typical messages from operational users and administrators and their necessary qualifications, reflected by the likelihood of errors and the quality of the information issued;

• possible negative and unauthorized impacts from the environment when using PS;

• the necessary characteristics of the computing facilities on which the complex of programs with the required quality is intended to function.

Comparing the results for When Evaluating the quality of Characteristics with the 'requirements of the Contents Technical specifications and specifications, the developer or supplier is Obliged to Satisfy the' requirements of the customers the only Within the parameters The Agreed of the model is environmental. It was agreed upon by the testers with the developer. In the case of the patient, he may be qualified to refine the programs to meet these requirements.

The Internal qualification tests of software quality (tests of the chief designer), the which are often the combined with the completion of complex debugging, a must the BE Documented and The form the basis upon Presentation of the PS to the to customer for qualification tests for final Evaluation Download now of the quality Characteristics of the software product (see ISO 12207, ISO 15504, ISO 16326). The developer must implement and evaluate the project

423

Lecture 14. Integration, qualification testing and testing of complexes ...

grams, tests, test results and user, given:

• it has been confirmed that it has been approved to

• consistency with the required customer and the expected results of the use of PS;

• possibility of integration and testing of PS in the system;

• the ability to operate and maintain versions of PS in accordance with the requirements of the contract.

Any tests are limited by the permissible number and volume of inspections, as well as the duration of the work of the commission of testers, therefore, they cannot guarantee absolute quality control of the product. To increase the reliability of determining and improving the evaluation of PS characteristics after internal testing, it is advisable to transfer the set of programs to some users for trial operation under typical conditions.This allows you to more deeply assess the performance characteristics of the created complex and eliminate some defects and errors. It is advisable to conduct trial operation by developers with the participation of customer testers and some users appointed by the customer. The results and characteristics of the quality of trial operation after testing by the chief designer can be taken into account when the customer conducts qualification tests for their reduction.

If you want to go through the system of It didn’t take it in accordance with the standards and rules. In the case of the system, it has been established that it has been approved in accordance with ISO 12207, ISO 15504. stages of the implementation of qualification testing and testing (Fig. 14.2):

- qualification testing of the components outside the system;

424

14.2. Complex groups complexes

• integration of the system hardware;

• qualification testing and system software in conjunction with the software.

Software Qualification Testing I

I out of system:

- Testing the II program package; II - preparation for the system; II - assessment of the quality of use

software product to destination

i

I Integration and Testing

I system instruments:

I - tests for the t2 to the system; I

- assessment of feasibility and test planning of the program complex

I as part of the system; I

I - analysis of the completeness and correctness of the documentation for the program complex

Qualification testing of the system with a complex of programs: I

I - determination of conformity of system quality characteristics and

I gram product contract requirements and specifications; II - certification of the adequacy and quality of technological and operational

I documentation of the system and software; I

- execution of the certificate of completion and the contract for the creation of version II of the software product and system I

I Documentation supporting qualification testing and I testing a set of programs:

• requirements of the contract, technical specifications and specifications for the I system and software product;

• profile and set of standards and regulatory documents;

• Test program of the complex programs;

I - test procedures for compliance with the requirements of the contract, specifications and test programs;

- test reports by sections of the Program;

- a set of adequate operational and technological documentation; I

I - certificate of completion of tests and fulfillment of the terms of the contract for I

I system and software product

Fig. 14.2

425

Lecture 14. Integration, qualification testing and testing of complexes ...

Qualification testing of functional components and PS in generalperformed in order to demonstrate to the customer that all the requirements of the contract have been implemented and the necessary quality of functioning of the program package has been achieved. Qualification testing should cover all component requirements in the specification of requirements for the PS and in the specifications of requirements for interfaces. Testing the components for each configuration should show that the requirements for the components to be implemented in this configuration are fully met. Responsible for the qualification testing of components should not be the persons who carried out the implementation of the working draft or programming of the corresponding component. This does not exclude the possibility of assisting in the conduct of qualification testing by the persons performing the working draft or programs, for example,by providing test options based on their knowledge of the internal implementation of the component or PS.

The developer must determine and register the test preparation process, test cases, scenarios and procedures to be used for qualifying the component and trace the correspondence between test cases and contract requirements. He must prepare the test data necessary to carry out the test options and notify the customer representative in advance of the time and place of the qualification testing. If a component test is to be witnessed by a customer representative, prior to the test, the developer should check the test options and test procedures to ensure that they are complete and accurate and that the PS is ready for testing in the presence of the customer representative. The results of these checks should be recorded in the PS development files,and test options and procedures are appropriately modified to eliminate the identified defects. It is necessary to make all the necessary changes to the software, after notifying the customer’s representative, to re-test the required volume, modify the software development files and other software products, based on the results of the integration and testing of modules and components.

426

14.2. Organization and methods for evaluating the characteristics of complex program complexes

The results of these actions should be included in the report for the customer on the preliminary tests conducted by the developer.

Integration and testing of PS in the system’s hardware contains their integration, testing of the resulting complex to determine whether they work together as required by contract, and should continue this process until integration and testing is completed for all components. programs and equipment. Test cases should cover all aspects of the system level and the required quality characteristics of the project. The developer must make the necessary adjustments to the programs, take part in the re-testing in the required volume and modify the PS development files and other software components, based on the results of integration testing.

Qualification testing of the system and software in generalThe contractor’s contract was not valid. It should be noted. Tests should be approved by the customer representative. It’s not a problem, but it’s not a problem.It can be carried out. For each requirement, it needs to be traced in the system requirements. It has been given that it is necessary to ensure that it is not a problem. for subsystems Must be chosen,It has been given that it is necessary to ensure that it can be used. the system requirements. It is a requirement to ensure that it is not a problem.It is a requirement to ensure that it is not a problem.It is a requirement to ensure that it is not a problem.

427

Lecture 14. Integration, qualification testing and testing of complexes ...

All results obtained should be included in the test report of the PS and the system. If the qualification testing of a system is to be certified by a customer representative, then before it is carried out, the developer should check the test options and test procedures to ensure that they are complete and accurate and that the system is ready for testing in the presence of the customer representative. Tests should be performed in accordance with customer-approved test options, scenarios and procedures.

The evaluation of the quality of a software product under qualification, acceptance tests is carried out by the customer’s commission, in which the head (chief designer) of the development and some leading developers or a certified certification laboratory participate. The commission at tests should be guided by the following documents (see fig. 14.2):

- approved by the customer and agreed with the developer by the contract, technical specifications and specifications of requirements on the PS;

• current state and departmental standards for the life cycle and testing of programs, technological and operational documentation, as well as de facto standards agreed with the customer for use - a profile of standards and regulatory documents;

• Test program for all requirements of the contract, technical specifications and specifications;

• test methods covering each section of the requirements of the technical specifications, specifications and test programs;

• a set of adequate operational and technological documentation for a set of programs.

The test program is a plan for conducting a series of experiments and should be developed from the standpoint of permissible minimization of the amount of testing in the process of testing to verify that the requirements of the technical specification are met and that the documentation submitted is in compliance. The test program, methods of their conduct and evaluation of the results, developed jointly by the customer and the developer, must be agreed and approved. They should contain clarifications and detail requirements of the technical specifications and specifications for this PS, as well as to ensure correct verification of all specified

428

14.2. Organization and methods for evaluating the characteristics of complex program complexes

quality characteristics. The test program should contain the following clearly defined sections:

• the object of testing, its purpose and the list of basic documents that determined its development;

• the purpose of the test, indicating all the requirements of the technical specification, characteristics and quality attributes to be checked, and restrictions on testing;

• the Test Program itself, which contains a check of the completeness of the designed PS in accordance with the technical task and a test plan for checking all sections of the technical task and additional requirements formalized by individual solutions of the developers and the customer;

• test methods that uniquely define all concepts of verifiable quality characteristics, conditions and test scenarios, tools used for testing;

• methods of processing and evaluation of test results for each section of the Test Program.

The test method should contain: a description of the organization of the testing process, test cases, scenarios and procedures that are used in testing a single component or PS as a whole. Each test must have a unique identifier for this project; instructions for testing should be provided; description of equipment and PS for the implementation of testing, as well as instructions for re-testing. In addition, there should be references to the requirements to be checked, the conditions for implementation (configuration of equipment and components of the substation), input data, reference and expected results, criteria for evaluating the quality of results, the procedure for testing for each test case, assumptions and limitations.

The PS test plan should describe the order of qualification testing of components and subsystems, the test environment to be used during testing, identify the tests to be performed and indicate the schedule of test actions. For each proposed test implementation, the following should be specified: hardware versions used; interface equipment; additional external devices; test message generators; set up

429

Lecture 14. Integration, qualification testing and testing of complexes ...

test synchronization. In addition, the document should include a testing schedule and a test trace matrix for the requirements of specifications at the PS or its components, as well as subcontractors participating in testing, their role and responsibility.

The large amount of heterogeneous data obtained during testing of large-scale PS, and the variety of possible methods for their processing, interpretation and evaluation, lead to the fact that the most important factors are the methods of processing and evaluating results, as well as test protocols for testing program items. In accordance with the test methods, automation equipment should ensure the full completeness of performance tests for each section of the methods. The test results are recorded in the protocols, which usually contain the following sections:

• purpose of testing and the section of the requirements of the technical specification for which the tests were conducted;

• indication of sections of the methods in accordance with which the tests, processing and evaluation of results were carried out;

• conditions and scenarios for testing and characteristics of the source data;

• generalized test results with an assessment of their compliance with the requirements of the technical specifications and other guidance documents, as well as technical documentation;

• a description of the differences between test and real operational environments;

• a description of the defects and errors found and the recommended improvements in the tested PS;

• conclusions on the test results and on the compliance of the established PS or component with a specific section of the requirements of the technical specification and initial specifications.

The protocols for the entire test program are summarized in the act, as a result of which a conclusion is made about the compliance of the system with the requirements of the customer and the completion of work with a positive or negative result. When fulfilling all the requirements of the technical specification, the customer is obliged to accept a set of programs, and the work is considered completed.

The first basic version of PS should be subjected to the most complete and versatile testing. When testing the next modernized

430

14.2. Organization and methods for evaluating the characteristics of complex program complexes

Substation versions may have significant reductions in testing of approved reused components. However, comprehensive and final testing of each new version of the PS, as a rule, are carried out in full, guaranteeing verification of the fulfillment of all the requirements of the modified technical specification. In order to detect defects during the operation of serial samples, each of them should have a certain minimum of means for verifying operation and detecting distortion of results. These tools should allow to fix the conditions of the program malfunctioning and the nature of the manifestation of defects when using PS. Subsequent correction of errors should be carried out by specialists involved in maintenance.

Prior to the beginning of the qualification tests, PSs shall be subject to verification and certification of the means ensuring obtaining reference data, means of simulating tests from external objects, means of fixing and processing test results. Qualification tests of the PS are completed by submitting to the customer for approval a set of documents containing the results of comprehensive testing of the software version:

• corrected texts of programs and data in the programming language and in the object code, as well as complete specifications of requirements for software components and software in general after the completion of testing and testing;

• PS test program for all technical requirements;

• a set of test methods and processing of results for all sections of the test program;

• tests, scenarios and test data generators used to test software and information components and the PS version as a whole;

• the results and protocols of qualification testing, functional and structural characteristics of PS in a real external environment;

• a report on the confirmation of the specified quality, the full characteristics of the achieved quality of functioning, as well as the degree of coverage by tests of the requirements specification for the PS;

431

Lecture 14. Integration, qualification testing and testing of complexes ...

• plan, methods and tools for automating customer and user training in the application of the tried and tested PS version;

• a set of operational documentation, a description of the PS and user manual in accordance with the terms of the contract;

• technical specifications for the PS version, database and operational documentation for replication and mass production;

• guidelines for the installation, generation of the user version of the software and loading of the database in accordance with the conditions and characteristics of the external environment;

• report on the technical and economic indicators of the completed draft version of the PS, the implementation of plans and resources used;

• certificate of completion of tests and readiness for delivery and / or presentation for certification tests of the PS version.

The above organization of testing large PS is focused on the presence of a specific customer of a complex of programs and a limited number of users controlled by the customer. Somewhat different is the testing of commercial software packages created by the initiative of a company or a team of developers for sale to a wide range of users in the absence of a specific customer. For such commercial program complexes, it is customary to conduct qualification tests for compliance with criteria formalized by the project manager in two successive stages - Alpha- and Beta-testing. They consist of normal and accelerated (stressful) trial operation by the end users of the completed software product in accordance with the operational documentation and differ in the number of participating users and their level of qualification.

Alpha testing involves end users working primarily in the same company, but not directly involved in the development of a complex of programs. For Beta testing, voluntary users (potential buyers) are attracted to whom the version of the software is free for trial use. At the same time, the participation of competent, interested and friendly users who are able to identify defects and, with their recommendations, improve the quality of the tested programs, is of particular importance. Their activities are stimulated by free and early receipt and development.

432

14.3. Means for testing and characterization of complex systems ...

new software product and its own assessment of its quality. These users undertake to provide developers with information on all identified defects and errors, as well as to make changes to the programs and data, or to replace versions only as directed by the developers. Only after successful operation and beta testing by a limited contingent of users, the project manager or development company can make a decision about transferring the PS for sale to a wide range of users. Summarized Beta testing results can be used as a basis for certification testing.

In Alpha and Beta trials, it is customary to separate progressive and regression testing. Progressive means testing new software components to identify defects and errors in source codes of programs and specifications. Regression testing is designed to control the quality and correctness of programs and data after making adjustments. The necessity and breadth of regression testing is determined by the fact that a significant proportion of changes after Alpha and Beta testing, in turn, contain defects and errors. The number of tests and the duration of both stages of testing are determined by expert developers or the project manager, depending on the complexity of the program complex and the intensity of the flow of changes.

14.3. Means for testing and characterization of complex program complexes

To ensure the high quality of large program complexes, appropriate problem-oriented integrated test automation systems are needed that can sufficiently replace test programs with real objects of the external environment. At the same time, the high cost and risk of testing with real objects almost always justify the significant costs of such integrated systems, if there are trials of critical software with high demands on the quality of program functioning, with a long life cycle and many developing versions. Tool

433

Lecture 14. Integration, qualification testing and testing of complexes ...

Easy tools to automate testing and program testing should provide:

• test definition - implementation of the testing process by the developer: input of test sets; test data generation; input of expected, reference results;

• execution of the section of the program under test between checkpoints for which the test tool can intercept operator input (keyboards, mice, etc.) and for which the entered data can be edited and included in subsequent test suites;

• management of tests and the program area for which the test tool can automatically execute test suites;

• analysis and processing of test results - the ability of the test tool to automatically analyze test results: comparison of expected and real results; file comparison; statistical processing of results;

• analysis of the coverage of source code tests for the detection of: operators that were not executed; procedures that have not been called; variables that were not accessed;

• analysis of the program's performance when it is executed: CPU load; load memory; references to specified data elements and / or code segments; the temporal characteristics of the functioning of the test program;

• simulation of the external environment - support of the testing process with the help of the model of data imitation from the external components of the information system

When creating test environment generators, two fundamentally different approaches are used, which can be conventionally called integral and differential. With an integral or empirical-statistical approach, the basis is a formal description of the input and output information of the simulated object, as well as the functional relationship between the data at its input and output. At the same time, the structure of the object and the processes implemented during the actual functioning of its components are irrelevant and not modeled. Baseline data and characteristics for constructing such test generators are obtained in field experiments or in the study of more detailed differential models.

434

14.3. Means for testing and characterization of complex systems ...

Differential or simulation models of test generators are based on descriptions of the internal processes of functioning of the components of the modeling object, its structure and interaction of the components. The results of the functioning of such models are determined by the adequacy of knowledge about the components and their characteristics, as well as their interrelations. This requires sufficiently detailed information on all the processes of the functioning of the components of the environment, which, in turn, may require even deeper modeling of their components.

In contrast to the full-scale experiment, the simulation of the external environment and tests on a computer has great potential for monitoring both the initial data and all intermediate and output results of the test object. In real systems, a number of components are sometimes unavailable for monitoring their state, since either it is impossible to place the meters of monitored signals into real subsystems to be tested, or this is associated with changes in the characteristics of the analyzed object itself. The advantage of the simulation of the external environment on a computer is also the repeatability of the results of functioning and the possibility of studying a large number of test options and scenarios. In contrast, field experiments are often impossible to stop at some intermediate phase or to repeat with exactly the same initial data.