Automated software testing: advantages and disadvantages

Our last blogposts on software testing were more about testing in general and abouttesting and AI.

Today we are focusing on automated tests.

The importance of software testing

Software testing is part of the development process and is used to check the software for various defects and eliminate them. There are a variety of helpful tools with which testing can be carried out manually or automatically.

Software testing is an important phase in the entire software development cycle and should be carried out continuously; ideally, software testers should be integrated into the development process right from the start. Thanks to their experience, common errors in development can be avoided directly.

Only if the customer receives a solution that is largely free of bugs, is perfectly usable from a usability point of view and no failures are to be expected, will he be satisfied with the software. For this reason, software testers and developers work closely together at VIALUTIONS.

The difference between manual and automated testing

In manual testing, people carry out the tests themselves step by step without the use of automation technologies, whereas in automated testing, tools and scripts are used to carry out tests repeatedly and automatically.

This allows, for example, the monotonous and repetitive work of “clicking through” a software or app to be done by the computer. Instead of the tester opening the software themselves, e.g. logging in or opening a website, this is done by the respective tool, which in turn follows a list of predefined instructions. Examples of such instructions could be: “Find the login button.”, “Enter password xy.”, “Click on the login button” etc.

Advantages of automated testing

The major advantages of automated tests are speed, efficiency and accuracy. Our tester Bernhard sums it up as follows:

“The computer never gets tired of clicking through the same steps because it hasn’t had a coffee yet. It also completes most tasks much faster than a human could.”

Testers can get tired or unfocused, which can lead to overlooked errors or inconsistent results. In addition, automation technologies make it possible to carry out tests with hundreds of samples instead of just a few. This leads to a big gain in scaling. Automation is particularly suitable for regression tests, where the same test cases have to be executed again and again. Automated tests can also be executed on multiple platforms, devices and browsers simultaneously, which would be very time-consuming to do manually.

Disadvantages of automated testing

Disadvantages can arise with the accuracy of the tests. For example, if you want to test a SharePoint page, the entire test is dependent on the external service and its speed. If the service takes arbitrarily longer to respond, automated tests can be aborted unnecessarily for no apparent reason. Tolerances for the tests then have to be adjusted, which in turn has a negative impact on speed.

Another limitation arises in terms of coverage: not all test cases can be reasonably automated, e.g. usability tests or tests that require human intuition. Manual testers can react spontaneously to changes or unexpected software behavior and perform exploratory tests. For small projects or new software that changes quickly, manual testing can be a viable and cost-effective solution.

In other words, automated testing is very useful for positive test cases that prove functionality (“happy path testing”), but negative test cases are more difficult to map.

Challenges in automated testing

The most important thing about automated testing is that everything that is tested is deterministic. For example, if you run through the same steps 20 times 1:1 in a fixed sequence, you will usually get the same result 20 times. This is usually a given with normal software, but with websites that depend on external services or certain types of games, this can be anything but a given.

There are also features that should be taken into consideration when choosing test software.
For example, it is very helpful for debugging the tests if the software allows the tester to rewind and fast-forward the completed test like a video “frame by frame” in order to see and understand why a test did not deliver the result that was expected.
However, this function inevitably requires the software to take a “snapshot” of the entire page after each step so that it can restore the status at any point in the test and allow you to rewind and fast-forward.

This is not normally a problem when testing websites in isolation or individual components. However, if a page is dependent on external services and the test software does not offer an option to exclude their static content from the snapshots, it can happen that the required memory quickly exceeds the available memory and you are forced to deactivate this function.

Features such as these should be taken into account when selecting the test software, because just because a test software is “industry standard” does not necessarily mean that it is suitable for every use case.

Because at the end of the day, it doesn’t help the testers if the software has a 50/50 chance of failing during testing simply because the memory usage has been exceeded again. In this case, an alternative that does not involve “back and forth” with guaranteed stability would be a justified choice.

Conclusion

Automated testing is particularly suitable for large, complex projects that require frequent regression tests and fast release cycles. Manual testing, on the other hand, is better suited to smaller projects, exploratory testing or areas where human intuition and experience are important. In practice, many teams use a combination of both approaches to maximize the benefits of both methods.

You want to know more? Do not hesitate to contact us!