Automated Functional Testing

Lecture



Automated Testing Software ( Automation Testing )
- is the process of software verification, in which the basic functions and steps of the test, such as running, initializing, executing, analyzing and issuing the result, are performed automatically with the help of tools for automated testing.

Automated Functional Testing Software ( Functional Automation Testing )
- This is the process of verifying the functional requirements and features of the application under test, using the tools for automated testing. (see also Functional Testing)

We offer your attention to the following topics, which give answers to the most common questions that arise during the implementation of automated testing :

  • Why automate?
  • What to automate?
  • How to automate?
  • Tool selection
  • Test automation levels
  • Test Architecture
  • Strategy for using automated tests

We also recommend reading the articles of automated testing specialists :

  • We write a system of automated tests from scratch (
  • Page Objects pattern or working on science
  • Page Objects pattern + Selenium (Java)
  • Automated testing: working with static resources
  • Starting Selenium Server

Why do you need to automate?

A lot of misconceptions are associated with the automation of testing, as well as with many of the narrowly focused IT disciplines. In order to avoid inefficient use of automation, it is necessary to bypass its disadvantages and maximize the benefits. Next, we will list and give a brief description for the main nuances of automation and give an answer to the main question of this article - when automation should be used.

Benefits of test automation:

  • Repeatability - all written tests will always be executed in the same way, that is, the “human factor” is excluded. The tester will not miss the test of carelessness and will not confuse anything in the results.
  • Fast execution - the automated script does not need to be checked with the instructions and documentation, it greatly saves execution time.
  • Lower support costs - when automated scripts are already written, it usually takes less time to support them and analyze the results than to conduct the same amount of testing manually.
  • Reports - automatically sent and saved reports on test results.
  • Execute without intervention — during the execution of tests, the tester may be engaged in other useful tasks, or tests may be performed during off hours (this method is preferable since the load on the local networks is reduced at night).

Disadvantages of test automation (there are also a lot of them):

  • Repeatability - all written tests will always be executed in the same way. This is also a disadvantage, as the tester, performing a manual test, can pay attention to some details and, after performing several additional operations, find the defect. The script can not do this.
  • The cost of support - despite the fact that in the case of automated tests, they are less than the cost of manual testing of the same functionality - they are still there. The more often the application changes, the higher they are.
  • High development costs — the development of automated tests is a complex process, since an application that actually tests another application is actually being developed. In complex automated tests, there are also frameworks, utilities, libraries, and more. Naturally, all this needs to be tested and debugged, and this takes time.
  • The cost of a tool for automation - in the case of using licensed software, its cost can be quite high. Freely distributed tools are usually more modest in functionality and less user friendly.
  • Skipping minor errors - the automatic script may skip minor errors to check which it is not programmed. This may be inaccuracies in the positioning of windows, errors in the inscriptions that are not checked, errors of controls and forms with which the interaction does not take place during the execution of the script.

In order to decide whether to automate an application, you need to answer the question “do the benefits in our case outweigh the benefits?” - at least for some of the functionality of our application. If you cannot find such parts, or defects in your case are unacceptable - you should refrain from automation.

When making a decision, it is worth remembering that the alternative is manual testing, which has its drawbacks.

In the search for effective places to automate you can help the chapter "What you need to automate."

What you need to automate?

Asking: “ What to automate? ", You must first answer the question:" Is it expedient to automate testing under project conditions? " If the answer is "YES", then it is necessary, based on the requirements for the test object, to create a plan according to which automated tests will be developed. When creating such a document, you must clearly understand “ what to automate? ”, “How to automate?” and "how to automate?". We will not go into details on how and how to test this or that function, we simply list the places where , in our opinion, automation should be used :

  1. Hard to reach places in the system (backend processes, file logging, writing to the database)
  2. Frequently used functionality, the risks from errors are quite high. By automating the verification of critical functionality, you can ensure that errors are found quickly, and hence their quick solution.
  3. Routine operations, such as enumeration of data (forms with a large number of input fields. Automate filling in fields with various data and checking them after saving)
  4. Validation messages (Automate filling in fields with incorrect data and checking for the appearance of one or another validation)
  5. Long end-to-end scripts
  6. Verification of data requiring accurate mathematical calculations
  7. Validation of data retrieval

And also, much more, depending on the requirements for the system under test and the capabilities of the chosen tool for testing.

For more efficient use of test automation, it is better to develop separate test cases checking:

  • The basic operations of creating / reading / modifying / deleting entities (the so-called CRUD operations are Create / Read / Update / Delete).
    Example : creating, deleting, viewing and changing user data.
  • Typical application scenarios, or individual actions.
    Example : a user visits a mailing site, scrolls through letters, browses new ones, writes and sends an email, leaves the site. This is the so-called end-to-end script that tests a set of actions. We suggest that you use just such scenarios, since they allow you to return the system to the state as close as possible to the original one, which means that it has a minimal effect on other tests.
  • Interfaces, work with files and other moments that are inconvenient for manual testing.
    Example : the system creates some xml file whose structure needs to be checked.

This is the functionality, from the automation of testing which, you can get the greatest return.

How to automate?

In this section, we consider the aspects that influence the choice of a test automation tool.

First , you should note how well the automation tool recognizes the controls in your application. In the case when the elements are not recognized, it is worth looking for a plugin, or a corresponding module. If there is none, it is better to give up the tool. The more elements a tool can recognize, the more time you will save on writing and script support!

Secondly , you need to pay attention to how long it takes to support scripts written using the selected tool. To do this, write down a simple script that selects a menu item, and then imagine that the menu item you want to select has changed. If you have to overwrite the entire script to restore the script to its working capacity, the tool is not optimal, since real scenarios are much more complicated. Best of all is the tool that allows you to bring the name of the button to a variable at the beginning of the script and quickly replace its value. Ideally, describe the menu as a class.

And the last point to which you need to pay attention - how convenient is the tool for writing new scripts . How much time is required for this, how much can the code be structured (OOP support), how read is the code, how comfortable is the development environment for refactoring (processing the code), etc.

It is best to answer the questions “Why? What? How? ” In that order. This will help avoid wasting time spent on nerves and finances. On the other hand, you can get reliability, speed and quality !!!

Choosing a test automation tool

The choice of instrument often depends on the test object and requirements for test scenarios , since Testing tools cannot support absolutely all technologies used in application development. That is, the choice of the tool comes down to the banal method of trial and error. As a result, often we choose several tools to test the functions of the application. For example, we check GUI using Mercury WinRunner , backend processes using “java based test tools” or other tools. The main aspects of choosing a test automation tool are discussed in the "How to automate?" Section.

Consider the tools for automated functional testing from different manufacturers:

Company Tool
Hewlett-Packard (Mercury Interactive) QuickTest Professional, WinRunner
IBM Rational Rational Robot, Rational Functional Tester
Borland (Segue) Silktest
AutomatedQA Corp Testcomplete
Microsoft Microsoft VS 2005
Seleniumhq Selenium

I would also like to highlight a separate Java library for automated testing ( java based test tools and libraries ):

Tool Description
Selenium It is a set of different software tools.
Watij Watij (pronounced wattage) stands for Web Application Testing in Java. Watij is a pure Java API. This is a real browser. Currently Watij supports automating Internet Explorer on Windows only. Future plans to support others like Mozilla.
HtmlUnit HtmlUnit is a "browser for Java programs". It makes it possible for you to invoke pages, fill out forms, click links, etc ... It is quite a good javascript support (it’s really a good javascript) and it’s easy to use. It is typically used for testing purposes or to retrieve information from web sites. HtmlUnit is not a generic unit testing framework. It is specifically for the purposes of the testing framework such as JUnit or TestNG.
HttpUnit Writes in Java, emulates the browser, emulates javascript, basic JavaScript authentication, cookies and automatic page redirection, and allows you to emulate , tables, and links. When combined with a framework such as JUnit, it is easy to verify the functioning of a web site.
Jamaleon It can be easily used for technical and non-technical users alike. This is a group of keywords. It can be defined in Java and mapped to these keywords. It can be organized as a script. The test scripts are then used to test case documentation.
Junit JUnit is a simple framework to write repeatable tests. It is an architecture of unit testing framework.
Abbot Abbot is a simple framework for Java and GUIs. Facilitates generating user actions and examining component state. Supports recording and playback on any Java application.
Marathon You need to take care of your needs. The generated raw script can be re-factored for modules for efficient reuse and maintainability. Replay the scripts into the test suites.

There are also a huge number of frameworks and tools focused not only on Java, but also on other programming languages, such as: ruby, php, C #, javascript, python, perl, etc. We will review them in the near future.

Three levels of test automation

Conventionally, the application under test can be divided into 3 levels:

  • Unit Tests Layer
  • Functional Tests Layer (Non-UI)
  • GUI Tests Layer

To ensure the best quality of the product, it is recommended to automate all 3 levels. Let us consider in more detail the testing automation strategy based on a three-tier model:

Unit Test layer

Automated tests at this level are understood as Component or Unit tests written by developers. Testers no one forbids writing such tests that will check the code, of course, if their qualifications allow it. The presence of such tests in the early stages of the project, as well as their constant replenishment with new tests that check "bug fixes", will save the project from many serious problems.

Functional Test Layer non-ui functional level

As a rule, not all the business logic of an application can be tested through the GUI layer. This may be an implementation feature that hides business logic from users. For this reason, by agreement with the developers, for the testing team, access can be directly provided to the functional layer, which makes it possible to test the business logic of the application directly, bypassing the user interface.

Test level through user interface (GUI Test Layer)

At this level, it is possible to test not only the user interface, but also the functionality, performing operations that invoke the business logic of the application. From our point of view, this kind of end-to-end tests give a greater effect than just testing the functional layer, as we test the functionality, emulating the actions of the end user, through a graphical interface.

Test Tools Architecture

For the convenience of imposing automated tests on existing test cases, the structure of the test scripts should be similar to the structure of the test case - Precondition, Steps & Post Condition.

We get the rule that every test script should have:

  • Precondition
  • Steps (Test)
  • Post Condition

We list the main functions of the script:

  1. Precondition
    • Initialization of the application (for example, opening the main page, logging in under the test user, switching to the necessary part of the application and bringing the system to a state suitable for testing)
    • Initialization of test data
  2. Steps
    • Direct test
    • Entering data on the test result, with the obligatory preservation of the reasons for the failure and the steps taken by the test
  3. Post Condition
    • Deletion of unnecessary test data created during script execution
    • Correct application shutdown

It is also recommended to create a common library for handling errors and exceptions. For example:

  • PreConditionException
  • TestCaseException
  • PostConditionException

As a result, taking advantage of the above recommendations, you will have a common architecture for test scripts and scripts. Having reviewed the articles on the use of the PageObject pattern in the section "Articles and Practical Tips for Test Automation," you can implement your own framework for test automation via the GUI.

Strategy for using automated tests

To test automation gave the desired results, namely, reduced the time for software testing, the following is proposed:

  1. Writing tests should be done by “specially trained people” - specialists in automated testing (Software Automation Testers). After writing, the tests are transferred to the manual testing team, which already carries out their daily launch and analysis of the results. Thus, automated tests are also tested, and as a result, their reliability and viability increase.
  2. Written and debugged tests can also be transferred to the development team to debug new versions.
  3. The development team is recommended to carry out a daily assembly, with the run of all the written tests at all levels of test automation. And only after the new version begins to meet the quality criteria, install the new version on the QA platform.

Writing and approach to test automation depends on the application development process. Taking the RUP (Rational Unified Process), described on the pages of the blog "ProTesting", I can offer the following procedure, divided into phases:

  • Inception phase - the choice of an automation tool, depending on which decides whether ready-made developments (frameworks) will be used or everything will be written "from scratch".
  • Elaboration phase - writing tests for the main architecture (in the future, these tests will be used to receive the build - Build Verification Tests)
  • Construction phase - more detailed automation: critical functionality, regression checking, end-to-end scripts
  • Transition phase - preparation of tests for transmission to the customer (if required)

A more detailed overview of the use of RUP in software development can be found in the following document: RUP_bestpractices_TP026B.pdf

created: 2016-04-02
updated: 2022-01-17
132764



Rating 9 of 10. count vote: 2
Are you satisfied?:



Comments


To leave a comment
If you have any suggestion, idea, thanks or comment, feel free to write. We really value feedback and are glad to hear your opinion.
To reply

Quality Assurance

Terms: Quality Assurance