Whenever developers make even the slightest modifications to their code, it could result in unexpected outcomes, such as breaking functions completely unrelated to the change. Source code modification can be anything ranging from a simple bug fix to new feature enhancements. These changes could either introduce new bugs in the application or worse, open up ones that were previously fixed. The resulting outcome can be anywhere from minor inconveniences of application usage to drastic downtime during production.
Therefore, engineering teams perform regression tests to ensure that previously fixed bugs do not resurface, so as to avoid outcomes that disrupt the current functionality of the application or cause irrecoverable damages.
“But what does it have to do with application security?”
Ok, time for a quick 101 - Regression testing is essentially testing applications for bugs that were previously addressed in prior iterations before the next release. The main purpose of this is to ensure that defects that were fixed during previous releases, if regressed, are identified for the development team to fix in time. It is also done to make sure that your recent changes behave as intended, and the changes did not inadvertently cause problems in functions that otherwise worked correctly before the changes.
However, regression testing historically is largely restricted to the functional and performance aspects of an application. But, changes in source code do not only affect the functional aspects of the application, they also affect the security posture of it. Extending the concept of regression to security, what if previously identified automated and manual security vulnerabilities in an application are checked for regressions prior to release? Pretty cool right?
“Is Regression Testing really necessary?”
ONE - As companies move towards DevOps and Agile frameworks, applications go through major code changes during its short development life-cycle. Plugging in security regressions along with an automated tool-chain and existing QA automation framework, pumps up your application’s test coverage by leaps and bounds. The resultant effect makes the application more robust making it not just functionally operate in the manner it was meant to but also operate securely. This consequently minimises the risk of the application malfunctioning or experiencing downtime during production, thereby reducing business loss and saving a ton of man-hours needed to fix the bugs if detected in later stages.
TWO - Penetration Testers not just rely on automated tools to perform security sweeps, but also spend valuable man-hours in dissecting deep-rooted logic flaws that can never be unearthed by point-and-click tools. Now, imagine if these vulnerabilities were to be manually checked prior to every major release. Not only is this operationally inconvenient, but is an absolute waste of critical security bandwidth. Automated security regressions save bounts of critical man-hours of penetration testers that can be put to better use.
“If Regression testing is all so great and dandy, why isn’t it applied to application security?”
There are multiple reasons that deter companies from extending the concept of regression testing to security. I’ve got three :
Security was not considered as a priority: Traditionally, functional defects have been more observable and measurable than the security flaws of an application. Let’s face it - Realistically, how many companies out there would actually block a major release if there was a Medium severity bug that was found during a security sweep? If the functionality faces issues, the potential costs are measurable, hence can be directly addressed as a critical aspect of the application. Security, on the other hand, is measured based on risk. If there is a flaw in the application, the first thing one wonders is ‘what are the odds that someone will find it and the possibilities of it being exploited’. And, if it is unmeasurable, then why expend time and resource on something that may or may not affect the business operations of the company? Therefore, security is put on the back-burner. Since security wasn't a priority, “security regression”, and more so “automated security regression” wasn't something that companies thought of integrating into their testing methodologies.
Misconception that ‘Security testing can only be performed by experts’: Until recently, security testing was exclusively meant for the experts. The lack of tools and tactics (made available to developers) deterred them from even thinking about applying security during development, and justifiably so. Application testing too, was largely a manual activity (restricted to a penetration tester’s machine) and required a lot of expertise and man hours to thoroughly test an application for security flaws. Hence, to integrate a new concept like security regression, it definitely required more effort and manpower.
However, things have changed. Now there is automation technology available (such as Robot framework) that can be used and extended to run security regressions. A lot of heavy lifting can be done these automation technologies. In addition, you can even write scripts that specify the parameters and test-cases to run automatically as part of your existing CI/CD testing framework. What’s more amazing is that one could potentially re-use the entire stack of functional automation scripts to be used in conjunction with a DAST automation pipeline. By utilising these tools and automation frameworks, you are not only improving the security aspect of your application but also reducing the pressure currently placed on the security teams with minimal additional effort.
Bridge between QA and Security: Despite the availability of tools and technology, QA Engineering teams still need a certain level of understanding (knowledge and skill) of how application security works to be able to perform security regression. While they can utilise their functional regression knowledge for certain security automation scenarios, they still need to understand some key aspects of security such as threat modeling, how exploit scenarios work, and how to integrate security tools into CI/CD pipeline.
While at first glance this may seem completely unrelated, trust me, I’m getting somewhere with this! In order to write security regression test-cases, QA engineers need to first understand how threat modeling works, and how to use it to write test cases. This knowledge can be applied to write security test cases for the purpose of integrating it into the CI/CD pipeline. QA engineers also need to understand how security testing scenarios work, at least on a conceptual level. This will help them understand how they can use security exploit scripts and walk-through scripts for security regression testing.
Furthermore, they require an understanding of how tooling works in the CI/CD pipeline. As much as your security vendors would love to help you with integrating regression testing into your testing frame so that you can plug-and-play, it isn't always reliable. Even a minor error may require you to reach out to your vendor repeatedly. With the understanding of tooling and how to integrate it into the CI framework, your QA engineers can change and adapt security regression testing as they see fits per your application development strategies.
It’s about time you used Security Regression
Fixing a security bug during production stage is ten times costlier than fixing it during UAT (SANS, 2017). By using security regression, security vulnerabilities can be found during the development cycle, thus enabling developers to fix them much earlier in the release cycle. This reduces the business impact of the application and greatly reduces the cost of fixing them on production as opposed to fixing them in the development cycle. Due to the potential benefits from security regression testing, and the current availability of tools and automated frameworks, integrating security regression into your CI/CD pipeline is far more beneficial than to leave it to the experts or worse, ignoring it.
Orchestron’s role in Security Regression
With Orchestron’s Client component, security engineers can run authenticated, parameterised DAST scans using their existing functional automation scripts. This enables security teams to uncover vulnerabilities that cannot be found by simply running DAST scanners in a “plain - vanilla” fashion. For exploits that are manually uncovered by penetration testers, such as logic flaws, exploit scripts can be designed to raise a bug within Orchestron along with details such as CWE, Vulnerability Type, Vulnerability Summary etc. Orchestron then correlates results from manual and automated sources to present per-regression test reports that enable engineering teams to assess how much an application has regressed from its preceding version.