Find it and fix it. It seems like a simple formula for protecting software applications from the increasing scourge of nefarious attackers.
In actuality, finding a single vulnerability can be quite difficult. First, you must realize that the vulnerability itself has already snuck through usability testing and potentially through security code review/audit. That means that, for all intents and purposes, your security teams must find something that shouldn't exist-and even if they locate a new vulnerability, you must determine if the millions of lines of code currently used by the dozens of applications you rely on are vulnerable.
Even after identifying a vulnerability, fixing it can be a daunting task that might take weeks or months to accomplish. What do you do in the meantime? Do you shut down the application or do you take your chances and continue running the application in the hope that the problem will be fixed before the vulnerability is exploited?
Find it and fix it. It's a lot harder than it sounds.
Part One: Find it
According to the WhiteHat Website Security Statistics Report,1 82 percent of websites have had a high, critical or urgent issue and 63 percent of all Websites currently have a high, critical, or urgent issue (2009). Finding a vulnerability has two very distinct challenges: knowledge of vulnerabilities and continuity of identification.
Knowledge of Web application vulnerabilities-including how they work, how to detect them, and how to mitigate them-is not a commodity in general terms. The number of security analysts who can reliably and consistently apply this knowledge is far outstripped by the number of businesses, applications, and lines of code in use around the world, and not all of those with this knowledge are working for the good guys. The logical solution to this was to attempt to codify (literally) the knowledge of these few experts into applications that could quickly and efficiently scan code and applications to identify potential problems. The result was a much quicker way to identify hundreds or even thousands of possible issues. But, because these solutions were necessarily generic in design, they could not take into account the nuances of custom code-unique business logic-or specific implementations that require human interaction, yielding a significant amount of false positives. While these solutions provided a high-level triage of potential issues, it still required a skilled and knowledgeable information security professional to filter through the results and determine their applicability to the actual applications. This made the security analyst more productive, but did nothing to ensure that a sufficient number of skilled security analysts were active in the industry.
A second issue is that of continuity-ensuring the same, consistent security analysis across all web applications on a routine basis. As previously implied, applications that may have been deemed "safe" when deployed, may be susceptible to newly discovered vulnerabilities. Comprehensive security analysis requires a consistent and continuous process of examining both new applications as well as existing applications. This process further stretches the resources of the security analyst because each new application increases the workload exponentially, and any tangible relief offered by vulnerability scanners quickly disappears when the number of applications increases.
Part Two: Fix It
"Once vulnerabilities are identified it does not necessarily mean they are fixed quickly, or ever ... Still, resources and security interest are not infinite so some issues will remain unresolved for extended periods of time" (WhiteHat Website Security Statistics Report, 2009). Even after a vulnerability has been identified, fixing it presents another issue-one of time. Knowing where a vulnerability exists within the application only provides a clue as to where in the code the problem lies. It typically requires another team of experts to analyze the application and determine exactly where the code is deficient and how to fix it. Depending on the nature of the issue, even the best organizations may take weeks, even months, from the time a problem is identified until the problem is resolved. This is often because any code changes must be re-reviewed, regressed, and tested before they can be pushed into production. And that's just for one instance of a vulnerability, even though the same issue might exist in multiple places and in multiple applications. Just for example, WhiteHat reports that the average time to fix a single SQL injection vulnerability is 38 days. The average time to fix an insufficient authentication vulnerability is 125 days (WhiteHat Website Security Statistics Report, 2009).
Another issue with just fixing the problem is that many applications used within organizations are not developed in-house, but are custom developed, shrink-wrapped applications, or shrink-wrapped applications that are custom modified for each business. This can add significant delay in addressing the problem, even once it has been identified. And third-party applications, like shrink-wrapped applications, require the original developer to do their own analysis, regression, and testing internally before they even make a patch available. That patch then needs to be applied and tested by the organization before it can be deployed to production-often potentially doubling the time to resolution.
This delay in fixing the problem poses a serious dilemma for the organization. Vulnerabilities exist regardless of whether or not they are identified-and once one has been identified, the organization faces potential legal ramifications for not repairing it. Business owners have the choice of suspending the use of an application until it is fixed, or simply waiting and hoping that the application isn't exploited before a fix can be applied. Shutting off the application is rarely a viable option.
An Alternative Approach: Don't Find It; Just Fix It (Sort of)
Application firewalls provide hope as an alternative to the army of highly trained security analysts and developers required to find and fix website vulnerabilities. The simple idea, borrowed from the success of network firewalls, was to create a gate in front of all applications that only allowed identifiably "good" application interaction. The hope was that these devices could universally enforce policies that would mitigate potential vulnerabilities without identifying each one. While, in theory, application firewalls can accomplish this goal, it is not quite as simple as originally thought; application firewalls have some similar challenges.
For example, it is very simple to create an application firewall rule that blocks malicious traffic from reaching the application. But it is not as simple to create such a rule that won't also block valid user traffic. While the application firewall can enforce rules universally, each application has unique requirements. So, just like generic application scanners have difficulty identifying true vulnerabilities without the assistance of knowledgeable security analysts, application firewalls require knowledgeable security analysts to appropriately construct and maintain the policy (which takes a significant amount of time as well).
Another issue with deploying an application firewall by itself is that an organization typically has very limited assurance that it's actually protecting them from all the potential vulnerabilities that exist. Unless an organization is continuing to scan and test its applications, it will have no better knowledge of the security of its software with an application firewall than it does without one, as the firewall will only tell what attacks it did prevent. This means that although application firewalls provide an additional level of security, using an application firewall does not reduce the effort and knowledge needed to actually find vulnerabilities.
In addition, while an application firewall can protect an organization from having application vulnerabilities exploited, it does not actually fix the issues. Therefore, even if the application firewall policy is 100 percent perfect and the application never changes, the application is only "safe" as long as it remains behind the application firewall. If the application is ever accessed directly, it is completely vulnerable. The most appropriate course of action would be to use the application firewall as a temporary fail-safe, giving application developers more time to fix the issue without suspending the existing application or remaining vulnerable.
Find It AND Fix It
One attempt to improve the overall solution was to merge the potential vulnerabilities recognized by application security scanners with the application firewall. The scanner could identify all potential issues (real or not) and the firewall could protect against all potential issues. This approach combines the speed of the scanner identification (without the need for the army of highly trained security analysts) with the speed of protection provided by the application firewall-giving a much smaller team of highly-trained security analysts time to distinguish the real vulnerabilities from the false positives, and the development team time to fix the applications permanently. This solution has the best of both worlds: speed and ease of vulnerability identification and the speed and ease of instant remediation. Although an attractive option, this approach has also faced its share of challenges.
The first complication is finding a way to accurately describe the vulnerabilities so that the application firewall can accurately create a policy rule to mitigate the issue and provide an automated way to do so. Several vendors have created just as many schemas for the descriptions-most of which could only be generated by one vendor's scanner and understood by only one vendor's application firewall. The most successful attempt was probably the Application Vulnerability Database Language (AVDL), which became an Oasis open-standard but failed to find wide-spread industry adoption. Even with this, however, the process of exporting the vulnerability data from the scanner and then importing the data into the application firewall was labor-intensive and fraught with complications.
Another complication is that even with the ever-increasing processing power of today's servers and dedicated security appliance solutions, the unfiltered lists of "potential" vulnerabilities quickly created such massive application firewall policies that the performance of the application was severely impaired, often completely crashing the application firewall. Even the policy for a single application could quickly grow beyond the capability of a single firewall, not to mention trying to provide cross-application protection for multiple applications. The ease and speed of the solution quickly caused significant performance impacts and increased costs-two things that never help security adoption.
Analyzing the Problem
What have we learned about protecting web applications?
- Vulnerability scanners alone cannot provide web application security. They can only give you insight into the specific issues you have to fix. You still have to fix them and that takes time.
- Manual vulnerability identification is slow and expensive and there are simply not enough qualified security analysts to handle the load; particularly with thousands of custom and legacy applications that continue to change and evolve with the demands of modern business;
- Automated vulnerability identification alone cannot adequately identify issues with custom applications or unique business logic. Consequently, although they can quickly identify many potential vulnerabilities, they have an unacceptably high false-positive ratio that still requires manual intervention from qualified application security specialists.
- Application firewalls alone cannot provide web application security. They can block known or defined vulnerabilities precisely and generically block classes of potential vulnerabilities; they cannot provide assurance that you have blocked all vulnerabilities or that no vulnerabilities exist.
- Manual configuration takes significant amounts of time and requires qualified security analyst to do correctly;
- Requires some form of vulnerability assessment to keep policy up-to-date.
- A total solution involves both vulnerability identification and application firewalls. Periodic assessment of vulnerabilities provides assurance and application firewalls provide instant protection to enable an organization to adequately address the vulnerability.
- Assessment needs to identify quickly verified vulnerabilities (extremely low false positive ratio) on an automated and ongoing basis;
- Assessment needs to report verified vulnerabilities instantly to application firewall for mitigation;
- Application firewall needs to be able to implement specific, focused mitigation rules;
- Application firewall needs to be able to handle potentially thousands of such rules;
- Assessment needs to quickly validate mitigation and report the entire process.
Automatically Find It and Instantly Fix It
Fortunately, technology continues to evolve. The latest evolution in security is the idea of "as a service," as in Software as a Service (SaaS), Infrastructure as a Service (IaaS), and many similar solutions. The point behind these solutions is akin to outsourcing of the last decade: if you do not have the expertise and cannot afford or find the expertise full-time, outsource it to a specialist. The big difference is that, unlike outsourcing, these "-aaS" solutions are generally provided interactively via Web 2.0 services, much like object-oriented programming. You provide the input and receive the output.
There are now vendors who are offering the first Web Application Vulnerability Assessment "as a Service solutions." The input is simply the application and the output is a verified list of specific vulnerabilities. Unlike traditional automated scanners, these services provide additional vulnerability assessment coupled with human expertise that can accurately detect issues with custom applications and unique business logic. This amounts to outsourcing your application vulnerability assessment and validation process and addresses the problems without having dedicated security analysts or having to use scanners that report thousands of false positives. This solves the first issue.
This list of verified vulnerabilities is a significant step forward in the process of managing your application security and gets you quickly from "Am I vulnerable?" to "Yes, I am and here are the five specific vulnerabilities I have." This data still needs to be implemented within a web application firewall in order to provide the organization with the safety-net needed to provide the time to actually fix the vulnerabilities. Increasingly, these as-a-service solutions are not just providing a list, but programmatically interfacing with the application firewall to instantly push a fix as well. While this does require an application firewall that has an API interface to enable the integration and will not universally work with all application firewalls, those organizations who deploy application firewalls compatible with their assessment service, now have instantaneous capability to implement mitigation to specific vulnerabilities in real time.
Organizations can finally have the best of both worlds. First, by having access to a consistent vulnerability assessment that only reports actual, verified vulnerabilities, organizations can quickly prioritize and focus their development efforts without having to spend days or weeks ferreting out the false positives from the real issues. Second, by providing direct integration with application firewalls, organizations can instantly provide temporary mitigation of the issue without having to suspend application access or remain vulnerable until a fix is delivered. Lastly, because these services are only delivering actual vulnerabilities, the firewall policies remain trim and tight-ensuring that they don't inhibit application performance.
There is also significant benefit to organizations deploying their first application firewalls. Traditionally, this process has required an initial, base-line installation with very generic policies. Once that has been installed, the policies are tailored to meet the need of the specific applications-a process that can range from hours to months depending on the complexity of the application, how often it changes, and the level of security required. With these new offerings, organizations can deploy a new application firewall with the base-line installation. Then, using the vulnerability assessment service they can identify specific application vulnerabilities and instantly tailor the firewall policy for those issues. This enables the organization to implement extremely strong application security in a matter of minutes or hours, not weeks or months, regardless of the complexity of the application. And, if it changes, a simple re-assessment keeps them covered.
In the realm of web application security, the simple idea of "find it and fix it" has consistently eluded most, if not all, organizations. As the number of applications increases on a daily basis, the idea that we can manually scale to handle the issue is quickly losing credibility. For the same reason that we build applications to automate business, we need automated solutions to provide the security of those applications. With the advent of cloud computing and "as a service" offerings, a way to deliver consistent, on-demand, precise application vulnerability information with point-and-click remediation is finally within reach.
1. WhiteHat Website Security Statistic Report, 7th Edition. Santa Clara, CA: WhiteHat Security, Inc., 2009. Retrieved June 8th, 2009 from www.whitehatsec.com.
Introduction to Web Application Firewalls
About the Authors
Jeremiah Grossman, founder and Chief Technology Officer of WhiteHat Security and Ken Salchow, Technical Marketing Manager at F5 Networks.