Security Metrics Overview
Metrics is a term used to denote a measure based on a reference and involves at least two points: the measure and the reference. Security in its most basic meaning is the protection from or absence of danger. Literally, security metrics should tell us about the state or degree of safety relative to a reference point and what to do to avoid danger. Contemporary security metrics by and large fail to do so. They tell us little about the actual degree of "safety" of our systems or processes, much less about the organization as a whole. They say little about the appropriate course of action, and they are typically not specific to the needs of the recipient.
Clearly, there are designs and architectures as well as modes of operation and practices that generally result in safer operations than others. But unlike the Insurance Institute's crash rating tests for automobiles capable of predicting the outcomes of accidents in terms of injuries, there is nothing comparable for designing security systems or programs.
As with all other aspects of organizational activity, defining objectives for security is critical to determining an approach to getting there. It is also a requirement for developing meaningful metrics from both an operational standpoint and a strategic one. Without specific objectives to guide the direction for information security and to provide a reference point from which to measure, management will remain inconsistent, haphazard, and reactive. Providing those objectives and the "rules of engagement" is the function of information security governance.
The issue of security metrics has seen considerable activity in recent times, and there are numerous approaches to monitoring and measuring "security" available. However, the majority of these efforts generally apply to subsections of technical, or IT, security with a few notable exceptions. While these technical metrics are in many cases very effective at the specific task for which they were designed, they say little about the overall security, or safety, of the organization and provide little guidance for effective management. Measuring the state of "safety" of an organization is vastly broader than knowing how many packets a firewall dropped and is typically well beyond the scope of IT, or for that matter, information security.
If, in fact, the goal is to achieve meaningful "security metrics," then the approach to monitoring and measuring must strive to broaden its base to increasingly aggregate measurements from all the assurance functions an organization depends on to remain "safe." It would also be useful to develop a standardized set of metrics that could be generally applied using the same yardstick. Such a standardized set of security metric would have a set of required attributes such as being meaningful, actionable, consistent, and repeatable. However, even if a measure is well defined, the critical element is to track the measure across industries over time to determine what "31 inches of security" actually means in terms of probable costs, losses, and so on. As an example, the life insurance industry knows that an individual who smokes two packs of cigarettes a day, has high blood pressure, and lives in Los Angeles has a probable life expectancy that can be determined with a degree of accuracy at least on a statistical basis. No such correlations exist for security metrics.
Some instances of proprietary solutions in particular situations do exist where to a limited extent such correlations exist, but these correlations lack the depth and breadth for general application. Work continues on these tools by the private sector and governments. For example, SecurCompass is a proprietary security assessment tool that compares individual organizations to the averages for various industries using 500 metrics that are mapped against the security goals of executive management. The limitations of this approach are much the same as an audit. While perhaps more useful in some respects, it is still a snapshot in time as opposed to an ongoing real-time measure of organizational safety capable of capturing changes as they occur. It isn't a compass that can tell us to turn left or right. Other similar approaches exist but suffer the same limitations. One promising effort that is publicly available is the Metrics Center, which is being developed and managed by PlexLogic in conjunction with SecurityMetrics.org.
An effort to define objectives for technical security metrics suitable for management could, for example, be a dashboard that would show the results of an integrated system that monitored internal and external threats, system and process vulnerabilities, asset criticality and sensitivity, and the ongoing state of an organization's incident response capabilities simultaneously, and then present management with a real-time indicator of financial exposure. This hypothetical gauge would have a redline set at acceptable risk limits and a risk never to exceed (RNE, to coin a new acronym) mark that would be consistent with levels that would cause major harm to the organization. Obviously, the state of metrics is far from this objective, but it may nevertheless be useful to chart a direction for there to be any hope of achieving the goal.
Full audits and comprehensive risk assessments are typically the only activity organizations undertake that provides this breadth of perspective. While important and necessary, from a security management point of view, these provide only history or a snapshot, not what is needed in day-to-day security management. The 20-20 hindsight provided by audits also suffers from the assumption that a prudent path to the future can be paved with experiences of the past. In the dynamic world of security, with its ever-changing threat landscape, this is often not a safe assumption.
For some time, vendors have made efforts to integrate a variety of technical security indicators and to provide a "security dashboard." Significant progress has been made. For example, besides offerings from Computer Associates (CA Unicenter ), and IBM (IBM Tivoli), many others such as Intellitactics SAM are being offered.
A number of primarily technical data can be "rolled up" to present a real-time picture of technical security performance. Although still not yet widely deployed, these systems can be useful in managing the operations of IT security and can be combined with monitoring tools such as event correlation and tracking and SIM, to
some extent useful for security program management as well. Many current solutions such as ClearPoint Metrics go to lengths to present metrics in forms such as scorecards, and on a schedule that matches what a financial executive would expect.
All of this security metrics and compliance dashboard activity is a subset of the flurry of activity that is taking place in the measurement and reporting of organizational performance. In a discussion of the rise of compliance dashboards, Susan Jendrey quotes Michael Rasmussen, Vice President of Enterprise Risk and
Compliance Management at Forrester Research:
The dashboard provides a portal view into the state of compliance. Ultimately, the purpose of the compliance dashboard is to gather metrics and show measurement of compliance. It is a detection and reporting tool for things that can or have gone wrong.
If you ask any IT vendor if they have a compliance dashboard, the vast majority of them will step up and state that they do," muses Rasmussen. Early corporate adopters must be savvy in selecting a viable solution, since there is no single standard for information display, data integration support, and system architecture standards.
For example, CXO Systems' dashboard focuses on key IT risk indicators, which include compliance. There are a number of vendors building specific IT risk and compliance management dashboards, such as Archer Technologies, BindView, Hewlett-Packard, ITM Software, and Brabeion. There are specific SOX solution dashboards from vendors such as Certus and HandySoft. Then there are vendors such as Axentis, Paisley, Qumas, Open Pages, and IBM that provide broader enterprise risk and compliance dashboards, Rasmussen explains.
It is important to note that most of the work on dashboards has been rolling low-level metrics into higher level views. This does not necessarily result in something that executive management can use. As pointed out subsequently, this might be perfectly acceptable as long as the metrics are used to support business cases rather than as ends unto themselves.
Given the demonstrable benefits these systems can provide, the lack of greater penetration into enterprise information systems can be attributed to a lack of awareness, complexity, cost, and overhead. It is also likely that a persuasive business case has not been prepared nor a basis for computing return on investment developed. And it can be a significant job since dashboard agents must be deployed to all monitored systems and the data collected must be massaged and normalized in a way that allows meaningful integration. In addition, these efforts generally start from the wrong perspective in that they measure what they can, not necessarily what various recipients need.
As the importance of information security has become more apparent to senior management, the inability of current approaches to provide suitable "feedback" to effectively manage the plethora of required assurance functions has become increasingly clear. There is a growing consensus that security management technologies available today are insufficient for the needs of either executive or enterprise security managers. In part, the problem was recently stated by Shmuel Klinger, vice president of architecture and applied research in the CTO office at EMC, when speaking about security management:
I think in general we are on the completely wrong trajectory in management. Things are more complex, there are more moving parts, and management as an industry are chasing the wrong trends. These trends will have us falling on our face. We are increasing the amount of management data that we collect to a level of detail that no one cares about, which poses a nightmare for integration.
Another issue that must be considered is that in many organizations, the only way that security, or safety, issues are aggregated is by risk management and audit. But, the focus of risk management is not on performance or strategic alignment of security with business objectives; it is on identifying all sorts of risk and developing the controls or countermeasures to mitigate or manage it. Risk management is obviously important, but it is functionally different from both operational and strategic security management at the CISO or VP level. Audit is essential as well, but it provides only history, and it is hard to navigate with only a rearview mirror.
1. Metrics and Objectives
Metrics require objectives. Without defined objectives for an information security program it is not possible to develop useful metrics. It will not be possible to determine whether progress is being made or whether the program is headed in the right direction. Since we measure to manage, we must know what objectives we are managing to. In fact, most or all elements of information security governance must be implemented as a prerequisite for developing effective metrics. Without the underpinnings of governance-that is, the structure, rules, and processes to operate a security program toward defined objectives-it will be difficult to know what information is needed or, indeed, its relevance. Without clarity as to the destination, directional information, even if available, will be of little use.
Considerable high-level guidance for information security governance has been developed by the Information Security and Control Association (ISACA), which proposes six outcomes of information security governance and management:
- Strategic Alignment-Strategic alignment of information security in support of business objectives
- Risk Management-Executing appropriate measures to mitigate risks and reduce potential impacts on information resources to an acceptable level
- Business Process Assurance-Integration of all relevant assurance functions to maximize the effectiveness and efficiency of security activities
- Value Delivery-Optimizing security investments in support of business objectives to achieve the best return on security investments
- Resource Management-Using information security knowledge and infrastructure efficiently and effectively
- Performance Measurement-Monitoring and reporting on information security processes to ensure objectives are achieved
While few security practitioners would argue that these six objectives are important for IT and information security, the majority of organizations globally have made no effort nor are planning to implement metrics to track and manage achieving them.
This is dramatically highlighted by the recent IT Governance Global Status Report study of more than 7000 organizations. Table 1 from the global survey of IT and information security executives, shows the results of governance and metrics implementation for five of the aforementioned objectives and the utilization of ROI for IT.
The conclusion that must be drawn is that most senior management has yet to understand that like every other aspect of business, optimal and cost-effective security, or IT operations generally, cannot be attained without appropriate feedback mechanisms to gauge direction and performance. Surprising as these numbers are, it is nevertheless likely that as the cost of IT and security continues to increase and regulations become increasingly restrictive in the face of mounting losses from cybercrime, they will improve in the coming years.
Another ongoing issue is that while numerous studies over the years have shown the majority of losses (and therefore risk) to organizations comes from insiders, most security systems and their metrics still deal with external threats and establishing a secure "perimeter" after decades of advice from security practitioners to the contrary.
In a 2003 survey conducted by Harris Interactive Service Bureau and compiled by Vontu, a provider of software security solutions, noted the following key findings:
- 62 percent of survey respondents reported that incidents at work could put customer data at risk for identity theft.
- 66 percent said their coworkers, not hackers, pose the greatest risk to consumer privacy.
- 70 percent said that government regulations play a role in raising awareness at their workplace about identity theft and database security.
- Nearly 50 percent said that government still has not done enough to help thwart identity theft.
- 46 percent said it would be "easy" to "extremely easy" for workers to remove sensitive data from a corporate database.
If the results of this and other surveys are credible, greater emphasis must be placed on monitoring and metrics of internal activities. These results also indicate that controls and metrics other than technical ones will require more attention.
2. Information Security
To help gain clarity around the topic of security management metrics, the term information security needs a reasonably precise working definition. The word information has a meaning different from data and knowledge, although in common speech they are often used interchangeably. Information can be defined as "data with meaning and purpose." Knowledge can be defined as actionable information. Knowledge, in turn, is stored and disseminated as organized information.
We have already discussed that fundamentally, security is the assurance of safety. As a result we could define information security to include the assurance of the safety of data that has meaning and purpose and conclude that any other data is probably useless and a liability that needlessly consumes resources.
The purview of information security includes all aspects of information whether spoken, written, printed, electronic, or relegated to any other medium regardless of whether it is being created, modified, viewed, transported, stored, or destroyed. This is contrasted with IT security, which is concerned with security of information within the boundaries of the technology domain. Typically, confidential information disclosed in an elevator conversation or sent via regular mail would be outside the scope of IT security. However, from an information security perspective, the nature and type of compromise is not important, just the fact that security has been breached.
The IT Governance Institute defines the role of information security as "the protection of information assets against the risk of loss, operational discontinuity, misuse, unauthorized disclosure, inaccessibility, or damage. It is also concerned with the increasing potential for civil or legal liability that organizations face as a result of information inaccuracy and loss, or the absence of due care in its protection."
In the broader context of the organization, this definition constitutes a rather inclusive mandate typically beyond the scope of a typical security officer, and even a CISO. However, given the fact that according to a recent study by the Brookings Institution, intangible assets, such as knowledge, information, data, goodwill, patents, IP, etc., constitute 80 percent of the value to the typical organization today, the only surprise is the lack of integrated, concerted efforts to protect these assets consistent with a reasonable level of due care.
If "security" equates to the assurance of safety, the activities of security departments typically deal with only a subset of what makes an organization "safe." Other aspects of "safety" fall to a host of the other "assurance" providers and managers. For example, environmental safety may be the purview of facilities management, whereas product safety may be the responsibility of quality assurance. From a "security," or safety, standpoint, these activities, among many others, are highly relevant. In fact, when all organizational activities concerned with the assurance of safety are considered, they constitute a substantial component of all organizational activities and operational costs. A point of interest from an organizational structure perspective is that these assurance providers are generally not structured and governed by their strategic relevance or objectives but typically, by the operational processes they serve. The obvious example is that for most organizations, security is governed by IT, not by risk management.
3. IT Security
By definition, information technology security revolves around the machinery that processes, stores, and transports information. While IT security is concerned with the security of information within its boundaries, the focus is on technology. Yet, people and physical processes are inevitably interjected into technical processes at numerous points and typically represent the greatest risk of information compromise through accident, carelessness, ignorance, or intention. Technical controls and metrics certainly play an increasing part in catching mistakes, unauthorized access, and other threats to information security but can do little about social engineering, industrial espionage, carelessness, or fraudulent inputs.
Why the IT Metric Focus
This raises the question why IT security seems to get most of the attention while other assurance functions highly relevant to security do not. In part, it is due to the rapid evolution and recently realized level of dependence organizations have on IT systems in the face of ever more spectacular failures. Another factor is that other assurance functions have a longer history and more established and tested controls. Quality assurance, for example, has its modern roots in Deming's "zero defect" work over 50 years ago.
Another reason for the focus on IT security metrics is because they are relatively easy and can be automated. IT is machinery and lends itself to oil pressure and temperature gauges. The number of corporate secrets compromised at the local pub is far harder to get a handle on. Information security beyond the borders of technology is far more difficult to control and has been an issue since the birth of civilization, with encryption nearly as ancient. Roman couriers were concerned with it, and the famous World War II poster, "Loose lips sink ships" also deals with the subject.
4. Other Assurance Functions
The typical organization has a number of activities or departments that in some manner deal with safety, security, or risk management as contrasted with those that produce something. They often include:
- Information Security
- IT Security
- Physical Security
- Business Continuity/Disaster Recovery
- Human Resources
- Quality Assurance
- Risk Management
- Change Management
- Project Office
- Privacy Office
- Insurance Office
- Facilities Management
Admittedly, some of these devote only a portion of their efforts directly to safety, or security. Nevertheless, collectively these activities all have a role in "security," and are certainly relevant to organizational safety and risk management. From a management perspective, integrated reporting on a common basis from all these activities insofar as risk and security are concerned would be very desirable in providing a comprehensive picture of the overall "safety" of the organization.
In the past, management of the risk inherent in a business was a function embedded within the individual roles of the "C Suite." The traditional approach was to treat individual risks separately and assign responsibility to an individual or small team. Managing a singular kind of risk became a distinct job, and performing that job well meant focusing exclusively on that one particular area. The problem with this stovepiped approach is that it not only ignores the interdependence of many business risks but also suboptimizes the financing of total risk for an enterprise.
Breaking stovepipes and addressing the suboptimizing of investments requires a new way of thinking about the problem. This new thinking brings together the various stakeholders in the problem set to work closely together ...
There are probably better terms than security to denote the functions typically assigned to the department with that name. Arguably, it is one of many assurance functions that collectively look after the organization's safety and minimize its exposure to danger. Collectively, these functions are charged with preservation of the organization.
The relevance is that typical "security" metrics cannot of themselves, no matter how well conceived, provide much assurance of organizational safety. They are typically narrowly focused on the operational performance of specific technologies and generally serve only technical managers. These metrics fail to provide comfort to senior management that fraud will be prevented, that theft will be detected in a timely manner, and that someone won't physically steal technically "secure" servers.
The conclusion that can be drawn is that to measure and to report on the "security" of an organization and to provide the information needed for prudent strategic and management decision making, metrics must draw on a far broader set of data than is the current practice.
Any discussion of metrics must first and foremost consider the constituency. That is, who will be the recipients of metrics and monitoring feedback and reports; who will monitor, maintain, and calibrate the metrics; and so on? What information is needed by whom? The issue can be summed up by the question, "Who needs to know what when?"
A fundamental division exists between technical metrics necessary to operate and maintain the security machinery and management metrics needed to effectively and efficiently manage security and related activities. In some cases, metrics will clearly fall into one or the other category. In others, it is not as clear, and the metric might serve both, or as is frequently the case, neither.
Management metrics will be further subdivided depending on whether they are used to manage a security program or they are used to report an overview of the state of security to higher levels of management for strategic purposes. To reiterate, the critical component of metrics will be to determine who are the recipients and what information they require to discharge their responsibilities.
Getting Started with Security Metrics
In this audio interview, Krag Brotby explains the necessary preliminary steps you need to take before you start to collect data. It's a process of first determining the outcome, then the objectives to achieve that outcome, the strategies needed to reach the objectives, and finally the metrics needed to manage the process of achieving the outcome. As he makes clear, a security metrics program is much more than data collection and analysis.
About the Author
Krag Brotby has more than thirty years in the computer security field with a focus on governance and architecture. Brotby has served on the ISACA security practice development committee and has been appointed to the Test Enhancement Committee, which defines the practice area for the coming years. As a contributor of CISM examination questions, Krag has an intimate understanding of the type and level of security governance knowledge required to be successful at the examination. An early contributor to SABSA methodology and developer of the Business Process Assurance model (BPA) and the Rapid Security Assessment Model (RSAM), Krag has extensive experience with security governance issues and practices. He is currently focused on an information security metrics project for ISACA.
From Information Security Management Metrics by W. Krag Brotby, CISM. New York: Auerbach Publications, 2009.