Features testing installers

An installer is a "regular" program, the main functions of which are Installation (Installation), Update and Uninstall (Uninstall) software.

Proverbial wisdom is well known to everyone: “They meet according to their clothes, but they escort according to their minds” . The installation application is the very clothes that create the first impression of your product. That is why testing the installation is one of the most important tasks.

Being an ordinary program, the installer has a number of features, among which the following should be noted:

• Deep interaction with the operating system and its dependency (file system, registry, services and libraries)
• Compatibility of both native and third-party libraries, components or drivers, with different platforms
• Ease of use: intuitive interface, navigation, messages and tips
• Design and style of the installation application
• Compatibility of user settings and documents in different versions of the application
• And much more

If these features do not charge you seriously for testing installation programs, then I want to give a small list of risks that will show the full significance of the correct work of the installers :

• risk of loss of user data
• risk of operating system failure
• risk of application malfunction
• risk of incorrect application operation

At the same time, as with any program, some functional requirements are imposed on the installer . Combining them with a list of features, we get a more complete picture showing the amount of upcoming work on testing. And further, based on the list of requirements, you will need to answer the questions: "What to test in installation programs?", And only then - "How to test the Installation?".

In most cases, the installer is an application in the form of a wizard (Wizard), which may have specific requirements, recommendations for testing of which are discussed in the section: "Testing the installation wizard (Installation Wizard)"

With the modern abundance of personal computers, servers and operating systems, the need has arisen to install the same software on different platforms. To do this, installers need to understand what and where they install, depending on the environment. Some recommendations on this type of testing are discussed below in the section "Cross-platform testing of installers"

What to test in installation programs?

Let us write in more detail, "What?" you need to check to assess the correctness of the installer:

• Installation (Installation)
  1. Availability of sufficient resources for installation of the application, such as RAM, disk space, etc. (how to test ...):
  2. The correctness of the file list in the installation package (how to test ...):
     1. when choosing different types of installation or installation parameters, the list of files and paths to them may also differ
     2. no extra files (project files not included in the installation package should not be on the user's disk)
  3. Registering the application in the OS (how to test ...)
  4. Registering extensions for working with files: (how to test ...)
     1. for new extensions
     2. for already existing extensions
  5. Access rights of the user who puts the application: (how to test ...)
     1. registry rights
     2. Permissions to access files and folders, for example,% Windir% \ system32
  6. The correctness of the installation wizard (how to test ...)
  7. Installation of several applications for one run (how to test ...)
  8. Installing the same application in different working directories of the same workstation (how to test ...)
• Update
  1. The correctness of the file list, as well as the absence of extra files: (how to test ...)
     1. checking the file list with different installation parameters
     2. no extra files
  2. Backward compatibility of generated data (how to test ...)
     1. preservation and correct operation of data created before the update
     2. the ability of the old versions of the application to work correctly with the data created in the new versions
  3. Update while the application is running (how to test ...)
  4. Interrupting an update (how to test ...)
• Uninstall (Uninstall)
  1. Correct removal of the application: (how to test ...)
     1. removal of installed libraries and service records from the system registry
     2. deletion of physical files of the application
     3. delete / restore previous file associations
     4. preservation of files created while working with the application
  2. Uninstall while the application is running (how to test ...)
  3. Delete with limited access to the application folder (how to test ...)
  4. Deletion by the user without appropriate rights (how to test ...)
     
     

• How to test installations

  Taking advantage of the recommendations on "What to test in installation programs?", Consider " How to test installations? ".

  • Installation (Installation)
    1. The resources needed to install the software should be analyzed exactly at the beginning of the installation process. In case of their shortage, the user should receive an appropriate warning.
    2. Getting the list of files should be done before, and checking the files themselves - After installation!
       1. the most correct thing is to try to get a list of files from the installer of the installation package. The phrase: "Take and make a list after installing the software, everything is right there" - this is a provocation and it is better not to give in to it.
       2. if the program contains files signed by a certificate from MS, then it is possible that temporary files that are no longer needed for the operation of the application (* .tmp, etc.) may appear. Please note that the same installation package can install different sets of files for different OS. Therefore, it is necessary to test for each OS separately.
    3. Practically for any OS, registration of working libraries and service records is important. For example, in Windows, you will need to check the system registry for correct recording of new data and registration of a new / new application.
    4. After installing the application, you need to check the correctness of the registration of extensions for the working files of the installed program in the OS. Double-click on the document, as a result it should open with your application. If, as a result, you get a dialog to select a program for opening files of this type or it will be opened by another application, then there will be an error registering the extension in the OS.
    5. In an amicable way, the installation program should check the user account at the start and immediately report any problems with the rights. But the option is not excluded that additional restrictions will be placed on any service folders. You can try to simulate situations yourself when one of the folders is closed for writing, for example, "\ Program Files", "\ Windows", "% Windir% \ system32", and also to check how the application behaves when it is impossible to record any of files on the desired path. It is possible that without some files, the performance of the entire application will not be compromised. In this case, it is enough to indicate the problem with copying the file (s) to the log, and NOT to display an error message.
    6. It is necessary to conduct a full Testing wizard of the application (Installation Wizard)
    7. Quite often there are situations when the application in addition to itself puts another product. This can be either a separate third-party application, or a demonstration of its own product. During the installation process, it will usually not be another set of steps of the installation wizard, but simply a small message that such and such a product is being put. When testing, you will need to pay special attention to the fact that we have a consistent installation of several products. If the first one requires a reboot after its installation, then the second one may be installed incorrectly, especially if drivers are installed into the system both there and there. Of interest are the errors that occur at the junction of the installation of two applications.
    8. There are applications that can be installed in different working directories of the same workstation, and at the same time they will work independently from each other, without creating any conflict situations. But it's not always the case. There may be conflicts with access to shared resources on disk, in memory and / or in the system. All this should be tested thoroughly.
  • Update
    1. By analogy with paragraph 2 of the section "Installation (Installation)".
    2. Backward compatibility testing includes the following steps:
       1. after installing the update, all objects previously created by the application, such as documents, forms, save (if this is a game) should open and work without errors. This behavior is called backward compatibility. User settings should remain the same, of course, if the updates did not affect their change.
       2. single documents created in the new version should be correctly opened in older versions, of course, if the purpose of the update was not to change the format and structure of the files. If a new format has been implemented, then the new version should be able to save the document in the old format.
    3. In case the updated application is running, the user should receive a warning that the update is not possible while the application is running.
    4. Note that the update process can be stopped at any time, while the original application must remain unchanged and in working condition. Suppose you have installed version 1 of the application, you are trying to update it to version 2, but during the installation process, change your mind and interrupt the installation. In this case, the installer should return all changes already made, clean the temporary files used for the update, and complete its work. At the same time, the application version 1 remains in working condition.
  • Uninstall (Uninstall)
    1. It is considered a good practice to delete created during the installation process (registry entries in the registry, libraries in the% Windir% \ system32 system directories, files, etc.). Conventionally, the process of testing uninstallation can be divided into several parts:
       1. check that the system registry, service records and libraries installed in the installation process are deleted from the registry.
       2. checking the physical deletion of application files.
       3. check that after the removal of the application, the file extensions registered during installation are deleted, and the existing ones (registered before installation) are restored.
       4. checking the integrity of data created during the work with the application. It is likely that they lie somewhere in the depths of the catalog of the program itself. These can be service scripts, save from games or other user-created data, the deletion of which will cause damage to the user. Just assume that when you remove MS Office, all your documents will be deleted along with it. Therefore, such data cannot be deleted without user confirmation.
    2. If the application to be deleted is running, the user should receive a warning that the removal is not possible while the application is running.
    3. It is worth checking the behavior of the installer if the program directory is closed for deletion of access rights. In this case, the deletion process should not be performed, and the user should receive the appropriate message.
    4. If the user is not authorized to delete the application, he should receive the appropriate message and the removal process should be interrupted.

Testing the installation wizard or Installation Wizard Testing

Smart people wrote: "Wizards are evil." You can agree with this or not, but you still have to test them. The following test plan for installation wizard is proposed:

1. Identify all the paths from start to finish, and then prioritize each one. This will help us avoid unnecessary costs and effort when passing low-priority paths.
2. Forget the GUI. Try to describe test cases without binding to the interface elements. For example, a checkbox / radiobutton GUI control or a two-item menu is just a choice between true and false, it’s important what it ultimately affects.
3. If, as a result of passing the wizard, a file is obtained (a file describing the properties in the form of a list: property = value ), which is then passed on to the export procedure. In this case, you can divide the checks into two stages - the first, create (generate) such files and check that the export works correctly. The second is to check that the correct files are obtained through the GUI.
4. Do not forget to do this routine type of testing wizards, how to walk back and forth through the pages:
   1. without changing anything, all answers should be saved;
   2. changing something on the previous page, the next should be an adequate change or reset answers.
5. Make sure that the wizard responds adequately to the wrong answers and does not allow to go further.
6. The Cancel button (Close) should work always and on all pages of the wizard.
7. Create for each of the possible ways of the installation wizard a template result (ideally, make several of them - for different input data). Then, if possible, automatically or manually compare the result with the template result.
8. Highlight those options that do not affect any others, and which others do not affect. The operation of these options can be tested in isolation from others.

Cross-platform installer testing

I would like to single out a separate item cross-platform testing of installers , which must necessarily be carried out for all three functions - installation, update and removal:

1. The correctness of the installer with different versions of the OS, Service Pack (ServicePack) and installed updates
2. Checking files, drivers and libraries when installing under different operating systems
3. Verification of access rights to files, folders and system records for different operating systems
4. Checking the permissions set on the application files (Permissions)

To simplify the testing procedure, it is recommended to create a table where the columns will be the required configurations, and the lines - test cases (test cases) or test functions. In the process of testing at the intersection of columns and lines, fill in the result, which will be able to visually show the progress of testing and compliance with the "cross-platform" requirements. We now proceed directly to the description of the test:

1. If the application should work in several operating systems, the installer should check the necessary set of functions in the course of its work. It is possible that some kind of update package needs to be installed somewhere or you just need to work with another library.

   A case from practice : "The application must be closed for installation on Windows Vista. We launch it, the program thinks and gives a message that this version will not work under this OS. That's right, click OK. After that, the computer thinks for a couple of seconds and reports the following : "The application has been incorrectly completed. Let's try to run it in compatibility mode? "We agree and start. Our application automatically starts again, but this time it is put" with a bang. "We do not know what and how Vista is doing there, but we had to redo the protection code."

2. Checking the list of installed files is carried out by analogy with paragraph 2 of the section "Installation (Installation)", and as regards the installation of drivers and libraries, then the special importance of this check should be noted. Not all drivers and libraries work equally well on different platforms.

   A case of practice : "For some drivers, there is a dependency on the file system on which the OS runs. More precisely, for some types of errors. There was a case when everything worked fine on NTFS, and on FAT32 the OS could not rise after installing the software. The reason was incorrect write when you install the driver in the registry. "

3. It is necessary to verify that the installer has access rights to files, folders and system records. This is especially important when installing under an OS of the UNIX family, with their severe restrictions on access to resources for different categories of users. Also see paragraph 5 of the section "Installation (Installation)", as well as paragraph 4 of the section "Uninstall (Uninstall)"
4. When / after installation on unix systems, files must have appropriate permissions (Permissions). Those. if the file is intended to be launched, then it should be launched, if it is a configuration file, for example, then there should be permission for modification, etc.