First of all, let us see the meaning of the term Regression Test.
According to Wikipedia, “Regression testing is any type of software testing that seeks to discover software regressions.” Such regressions occur whenever software functionality that previously worked correctly stops working as intended. Typically, regressions occur as an unintended consequence of program changes. Common regression testing methods include rerunning previously run tests and checking to see if previously fixed bugs have resurfaced.
Final definition of “Regression Test”:
It is quite natural to have code changes during the software development and maintenance phase. Therefore, a software program that ran correctly before might stop working the same way after the changes. Therefore, the repetition of the test cases of a program, which worked correctly before the changes, and with the objective of detecting failures, can be concluded that it is a “Regression test”.
Regression testing is a quality control measure intended to ensure the following two conditions:
a) The newly modified code meets the specified requirements.
b) Unmodified code has not been affected by the above change.
By definition, regression testing is iterative. Therefore, most tests would be better suited to automation, where after performing a few test iterations, test automation would prove to be very cost-effective compared to the strenuous manual process.
Why are defects introduced during changes?
Apps that work properly fail due to incorrect or incomplete changes made to them. In the software industry, the rate of introducing defects into applications is quite high. In general, one in six attempts made on applications to fix them are faulty.
The main reason for the high rate of defects being introduced is
1) Poor documentation of the system held by the developers.
2) Tendency for developers to address symptoms instead of identifying underlying causes.
3) Lack of developer experience.
Objectives of regression tests:
1) The goal of regression testing is to identify unexpected defects. These defects or bugs are what remained in the system, because by changing the code, the developer probably could not fully understand the internal correlations of the code.
Therefore, regression testing remains the only reliable means of providing adequate confidence that changes or additions to the code are safe and cannot “break” the existing functionality of the application.
Therefore, regression testing becomes mandatory every time the code is changed or used in a new environment. It is essential to verify the integrity of the code in order to identify and correct defects as quickly as possible.
2) The goal of regression testing is not only limited to testing the correctness of an application, but also extends to monitoring the quality of its output. For example, when designing a compiler, regression testing can focus on code size, time to run the simulation, and time to run the test suites.
What are the best practices/strategies for regression testing?
1) Formulate a policy for regression testing on a regular basis, if we want to succeed in developing reliable and definitely predictable software applications.
2) Perform standard actions defined in the test procedure and verify that the desired responses are correct. Any failure of the system to meet the desired response set becomes a clear indicator of system regression.
3) Make sure the regression test is correct and not out of date.
4) Do a careful analysis of every defect that escapes detection during the regression testing process. This would necessitate careful updating of the regression test cases, so that such flaws don’t give us a slip in the future.
5) In general, unit test cases and integration test cases are used to build regression test cases; therefore, rather than having one large regression test, it is better to create a logical batch of such test cases in the form of a complete test suite. This would provide flexibility to run small unit tests in case of urgency or time pressures.
6) Following the famous 80/20 management principle, we can assume that there is a high probability that 20% of the functions will be used 80% of the time. Therefore, test suites for regression testing can be designed accordingly.
The severity of a problem in comparatively less common functions would be less if used by a small number of users. Therefore, the main focus of our regression tests should be on excessively used transactions, menus and screens, etc.
7) For smaller projects, do regression testing after every successful build or at least once a week. These repetitive activities can be automated with the help of tools like HP-QuickTest Professional etc.
8) Depending on the risk factors in the business, we can design the regression test suite. It has been seen that there are certain types of failures, which are not frequent, but when they do occur they have a serious impact on the business process.
Therefore, whenever any changes are made to the system or environment, we must perform regression tests that cover vulnerable areas that have a negative impact on the business.
9) We need to identify application areas that are known to have a high failure rate and include more regression testing on them.
10) Tightly tie regression testing to functional testing and build regression tests from successful test cases created and used during functional testing.
11) Periodic repetition of successful functional test cases (that verify the desired functionality of the application) as regression tests.
12) Regression testing should be like a chain, starting from the unit level, involving adaptation and repeating unit tests after detecting any changes in the unit. The regression testing chain continues through integration testing, system testing, user acceptance testing, and throughout the operational phases of the SDLC.
13) As a best practice, prior to releasing any mass build or in any live environment in your organization, you should run strict regression tests. Such an approach would help uncover major defects, if any, that could otherwise have serious implications on cost, productivity, schedule, and most importantly, adverse effect on the company’s reputation.
14) As a best practice, running regression tests at regular intervals should be an ongoing exercise in the case of all multi-user, web-based systems. Regular regression testing helps keep a continuous check on the performance of important transactions that are within specified limits.
Any factors responsible for slowing down the response time of any major transaction can be easily identified through frequent regression testing.
15) In addition to testing the functional aspects, we need to perform regression tests that cover the non-functional attributes of the application, such as its performance, security, usability, etc. The reason is that, in many cases, a minor change in code or design has been found to have a significant effect on system performance.
16) Before running time-consuming regression testing scenarios on newly delivered code, it is useful to perform smoke tests or sanity tests to save time, as smoke tests or sanity tests usually discover bugs obvious. Early detection of such errors can lead to early fixing.
17) We should be aware of the side effects of bug fixes. There is a high chance that the bug will be fixed, but at the same time our fix might have introduced another bug into the system.
18) Regression testing should be treated as an integral part of the extreme programming method of software development. This would involve extensive automated testing of the entire application at each stage during the SDLC.
Regression Test Results Errors
1) We should not be happy and satisfied just because the system has responded as desired, as there may be every chance of introduced defects being present, which could have escaped detection.
2) In most companies, critical application functionality is verified once and is assumed to continue to work fine until intentionally changed. However, the fact is that even minor code changes as a matter of routine can have serious unexpected side effects, possibly breaking previously verified functionality.
What are the benefits of having good regression testing policies?
1) Great improvement in the effectiveness of software development and testing staff.
2) Great success of development projects that resulted in reliable and stable applications.
3) Development teams modify the code without fear of breaking previously verified functionality.
4) Issues arising from code modifications are identified early in the lifecycle.
5) Huge savings in man-hours dedicated to finding and resolving software bugs introduced by code changes.
What are the attributes of a good regression testing policy?
1) Define solid guidelines for the use of the regression system.
2) Define concrete plans to consistently implement and integrate the guidelines defined in the SDLC. This results in the detection of existing bugs as well as newly introduced bugs due to unverified modifications made to the code over time.
3) Provides a mechanism to measure and monitor the implementation of the policy and the system to report the data that is tracked.
4) Solid and documented control and maintenance system for all types of regression test suites that are assets of great importance to the company. The policy outlines the system for backup, configuration management, past maintenance, and clear definition of ownership and responsibility of specific personnel for managing regression test suites.
What are the reasons why regression tests are not so popular?
Despite the many benefits of regression testing, many organizations do not practice it, likely for the following reasons.
1) Software development companies feel that regression testing is a bit complicated and difficult to maintain.
2) In most cases, companies do not have a well-defined and implemented policy towards regression testing.
3) Even if some policy is available, there is usually a lack of commitment from management towards the implementation of said policy.
