How to Measure Anything in Cybersecurity Risk by Douglas W. Hubbard and Richard Seiersen - A Book Review

Posted on November 9, 2024  •  27 minutes  • 5647 words

The Book and the Author

That book is based on the proposal of Douglas W. Hubbard’s “How to Measure Anything: Finding the Value of Intangibles in Business. ” Co-author of the book Richard Seiersen opens that proposal into the cybersecurity risk management field.

Douglas W. Hubbard is “the inventor of the Applied Information Economics method and founder of Hubbard Decision Research. His career is focused on applying Applied Information Economics to several business cases in corporations.

Richard Seiersen is a Chief Risk Officer at Resilience. In his career, he also worked as CISO of Twilio, GE Healthcare. He is also the author of “The Metric Manifesto: Confronting Security with Data”.

Their book proposes that cybersecurity risk management should not be based on ambiguous measurements. They first introduce the measurement as a concept. Throughout the chapters, they organize their proposal with real-world use cases.

Instead of having heat maps and ambiguous high-medium-low classifications, they propose having a quantitative approach. For instance, they propose to declare that instead of providing an argument like “the risk of data breach is high or medium for company X.” they suggest that: “the risk of data breach of company X in the next 12 months is %5.”

If, for example, we say that an event is 10% likely, do we mean there is a 10% chance of it happening sometime next year or sometime in the next decade? Obviously, these would be very different estimates. How is it that management or analysts could come to agree on these points when they haven’t even specified such a basic unit of measure in risk. (p.123)

I agree with their methodological proposal for cybersecurity risk management. The decision based on that research would be preferable, objective, and correct.

The book can be purchased on Amazon.  They also provided calculations of the methods explained in Excel template files on the How to Measure Anything website.

As a pro-quantitative professional, I want to thank Mr. Hubbard and Mr. Seiersen for their work. I want to cite some points from the book that took my attention.

Citations from the book

In this book, Doug and Richard continue to apply their passion to the topic of reducing uncertainty and making (much) better decisions in a profoundly complex problem space. As a cybersecurity professional for over thirty‐five years and a CISO for over 10 years, I can attest to how important this is. (p.14)

The bottom line is that managing cybersecurity well requires being able to identify and focus on what matters most. (p.16)

In other words, how do they make cybersecurity decisions to allocate limited resources in a fight against such uncertain and growing risks. (p.32)

How cybersecurity assesses risk, and how it determines how much it reduces risk, are the basis for determining where cybersecurity needs to prioritize the use of resources. (p.35)

For all practical decision‐making purposes, we need to treat measurement as observations that quantitatively reduce uncertainty. (p.40)

Measurement: A quantitatively expressed reduction of uncertainty based on one or more observations. (p.40)

A field called information theory was developed in the 1940s by Claude Shannon, an American electrical engineer and mathematician. In 1948, he published a paper titled “A Mathematical Theory of Communication,” which laid the foundation for information theory and, ultimately, much of the world of information technology that cybersecurity professionals work in. (p.41)

This “uncertainty reduction” point of view is what is critical to business. Major decisions made under a state of uncertainty—such as whether to approve large IT projects or new security controls—can be made better, even if just slightly, by reducing uncertainty. Sometimes even small uncertainty reductions can be worth millions of dollars. (p.41)

In 1946, the psychologist Stanley Smith Stevens wrote an article called “On the Theory of Scales and Measurement.” In it he describes four different scales of measurement: nominal, ordinal, interval, and ratio scales. (p.42)

…since this uncertainty can change as a result of observations, we treat uncertainty as a feature of the observer, not necessarily the thing being observed. (p.44)

So, there is a fundamental irony when someone in cybersecurity says they lack the data to assign probabilities. We use probability because we lack perfect information, not in spite of it. (p.45)

In the case of measuring cybersecurity risks, we are presumably conducting measurements to better allocate resources to reduce risks. (p.48)

The avoidably vague terms “threat capability” or “damage to reputation” or “customer confidence” seem immeasurable at first, perhaps, only because what they mean is not well understood. (p.48)

This became known as Laplace’s rule of succession (LRS). (p.55)

If you see 1 out of 78 major retailers suffered a major data breach last year, then LRS gives us a probability of (1 + 1)/(2 + 78), or 2.5% that a given retailer will suffer a major breach next year. (p.57)

Most experts are consistently overconfident in estimates and calibration training reduces this error. (p.69)

A Monte Carlo simulation uses a computer to generate a large number of scenarios based on probabilities for inputs. For each scenario, a specific value would be randomly generated for each of the unknown variables. Then these specific values would go into a formula to compute an output for that single scenario. (p.72)

The typical low/medium/high approach lacks the specificity to say that “seven lows and two mediums are riskier than one high” or “nine lows add up to one medium,” but this can be done with LECs. (p.84)

But once we have laid the groundwork, we could simply start with one arbitrary point and ask the following. (p.84)

[As an example:]

• Analyst: Would you accept a 10% chance, per year, of losing more than $5 million due to a cybersecurity risk?
• Executive: I prefer not to accept any risk.
• Analyst: Me too, but you accept risk right now in many areas. You could always spend more to reduce risks, but obviously there is a limit.
• Executive: True. I suppose I would be willing to accept a 10% chance per year of a $5 million loss or greater.
• Analyst: How about a 20% chance of losing more than $5 million in a year?
• Executive: That feels like pushing it. Let’s stick with 10%.
• Analyst: Great, 10% then. Now, how much of a chance would you be willing to accept for a much larger loss, like $50 million or more? Would you say even 1%?
• Executive: I think I’m more risk averse than that. I might accept a 1% chance per year of accepting a loss of $25 million or more… And so on. After plotting three or four points, we can interpolate the rest and give it to the executive for final approval. It is not a technically difficult process, but it is important to know how to respond to some potential questions or objections. (p.85)

Since we know at least one (if not both) must be wrong, then we know qualifications and expertise in cybersecurity alone are not sufficient to determine if a given opinion on this topic is correct. (p.88)

We propose that the single most important measurement in cybersecurity risk assessment, or any other risk assessment, is to measure how well the risk assessment methods themselves work. (p.88)

We assert that if firms are using cybersecurity risk‐analysis methods that cannot show a measurable improvement or, even worse, if they make risk assessment worse, then that is the single biggest risk in cybersecurity, and improving risk assessment will be the single most important risk management priority. (p.89)

The first principle is that you must not fool yourself, and you are the easiest person to fool. —Richard P. Feynman, Nobel Prize–Winning Physicist (p.89)

In perhaps the most ambitious study of this kind, Philip Tetlock conducted an experiment over a 20‐year period and published it in his book Expert Political Judgment: How Good Is It? The title indicates a particular focus in politics, but he interpreted it broadly to include economics, military affairs, technology, and more … poor performance by humans in forecasting and estimation tasks is partly due to inaccurate interpretations of probabilistic feedback. (p.96)

For example, if someone was not good at simple arithmetic, we wouldn’t be surprised if that person was not very good at estimating, say, the costs and duration of a large, complex engineering project with many interrelated elements (p.97)

They showed how even statistically sophisticated researchers will tend to greatly misestimate the odds that new data will confirm or contradict a previous experiment of a given sample size, and will incorrectly estimate expected variations in observations based on sample size. (p.97)

Unless we get regular, immediate, and unambiguous feedback, we are likely to have selective memory and interpret our experiences in the most flattering way. (p.98)

Do cybersecurity experts actually record all their estimates of probability and impact and then compare them to observation. (p.98)

if we say we lost reputation from a breach, how do we know that, and how did we actually validate—even approximately—the magnitude of the event as originally estimated. (p.98)

How many of your forecasts from five years ago do you recall accurately today? How many did you even document? (p.99)

In fact, our goal is to elevate the expert. We want to treat the cybersecurity expert as part of the risk assessment system. Like a race car or athlete, they need to be monitored and fine‐tuned for maximum performance. The expert is really a type of measurement instrument that can be “calibrated” to improve its output. (p.100)

Ultimately, testing subjective probabilities for calibration relative to overconfidence means waiting for observed outcomes to materialize. (p.104)

This approach and some of the research behind it were explained in the book The Wisdom of Crowds by James Surowiecki. Surowiecki also described several other collaboration methods, such as “prediction markets,” which show a measurable improvement over the estimates of individual experts. (p.109)

Some “expert aggregation” methods are consistently better than averaging and even better than the best individual expert. (p.109)

If we want to estimate the monetary impact of a denial‐of‐service attack on a given system, we can estimate the duration, the number of people affected, and the cost per unit of time per person affected. Once we have these estimates, however, we shouldn’t then just estimate the product of these values—we should compute the product. (p.110)

We must persist in the kind of skepticism that forces us to ask, “How do I know this works?” (p.112)

…only 13% of respondents say they use Monte Carlo simulations, and 14% say they use some form of Bayesian methods (although both of these are actually much more common responses than the authors expected). (p.119)

The International Organization for Standardization (ISO) standard 31010 states that the risk map (what Table A.1 of the standard refers to as a “consequence/probability matrix”) is “strongly applicable” for risk identification. (p.119)

Figure 5.1 shows the range of responses to a few of the terms covered in Kent’s study as they were reported in the book Psychology of Intelligence Analysis by Richard Heuer. (p.121) In that way, the phrase “Very likely” meant something different to the subjects when it was in the context of temperature extremes, glaciers melting, or sea level rise. (p.122)

…he has observed conversations about risks with clients where something was judged “highly likely” in part because of the impact it would have. (p.123)

A 10% chance per year is far too high for such a big event, so I think of 10% as highly likely.” This is purely anecdotal, of course, and we don’t rely on these sorts of observations. (p.123)

He has written extensively about the problems that ordinal scales introduce in risk assessment and how those scales are then converted into a risk matrix (which is then often converted into regions of “low” to “high” risk). He investigates all the less‐than‐obvious consequences of various forms of ordinal scales and risk matrices and how they can lead to decision‐making error. (p.125)

With this information Cox computes the “expected loss” (probability‐weighted loss), just as an actuary would do for many types of risks. He compares the products of the likelihoods and impacts of the risks: $200,000 for risk A (2% × $10 million) and $20 million for risk B (20% × $100 million). In other words, to an actuary, risk B would be considered to have 100 times the risk of risk A. (p.126)

Yet these two very different risks would actually be plotted in the same cell (that is, same row, same column) on a risk matrix. (p.127)

In this case, risk A has an expected loss of $4.5 million, and risk B has an expected loss of $1.2 million. Yet, if we followed the rules of this matrix, risk B is considered a “High” risk, and risk A is only “Medium.” This is what is called a “rank reversal” and it is not just an academic problem. (p.127)

But the ambiguity hides problems instead of facilitating the lack of information. Cox also points out that risk matrices ignore factors such as correlations between events. He stated in an interview with this book’s authors that “it is traditional to completely ignore correlations between vulnerability, consequence, and threat. Yet, the correlations can completely change the implications for risk management.” (p.127)

In short, matrices are ambiguity amplifiers. Cox summarizes his position for us by saying, “Simply on theoretical grounds there is no unambiguous way of coming up with such ratings in a risk matrix when the underlying severities are uncertain.” (p.128)

Thomas also looked at how various methods of categorizing likelihood and impact into a few discrete ordinal values (such as defining “Unlikely” as 1% to 25% or moderate impact as $100,000 to $1 million) modified risk rankings. Again, they found that these arbitrary design choices had a significant impact on ranking risks. (p.130)

Thomas found that any design of a risk matrix had “gross inconsistencies and arbitrariness” embedded within it. (p.130)

To drive home their finding, one of the PowerPoint slides in their presentations contained a large rectangular space titled “Heat Map Theory and Empirical Testing”. Showing a little humor combined with a lot of seriousness, the rectangle was empty. (p.130)

Given these problems, it seems clear that RMs should not be used for decisions of any consequence. (p.132)

As the great statistician George Box is often quoted as saying, “All models are wrong, but some are useful.” And to take it further, the research clearly shows that some models are measurably more useful than others. (p.133)

…statistical literacy is strongly correlated with acceptance of quantitative methods. (p.134)

Those who performed the worst on the stats literacy quiz were more likely to overestimate their skills in statistics. This is consistent with a phenomenon known as the Dunning‐Kruger effect. (p.136)

Those who believe quantitative methods are impractical in cybersecurity are not saying so because they know more about cybersecurity but because they know less about quantitative methods. (p.136)

From what we see of the research previously presented, not only do ordinal scales and risk matrices not correct for the errors of quantitative methods, they add errors of their own. (p.137)

Again, the previous research shows that ordinal scales and risk matrices might actually add error—that is, they literally reduce the limited information available to the intuition of the person using them. (p.138)

Even if one method has failures, it should be preferred over a method that has even more. (p.142)

Nothing is gained by the use of the popular scales and matrices. They avoid none of the issues offered as a challenge to more quantitative methods (complexity of cybersecurity, human agents, changing technology, etc.). (p.143)

There is nothing modeled with the qualitative scales that can’t be modeled with quantitative, probabilistic methods, even if we use. (p.144)

…only the same source of data as most qualitative methods (i.e., the cybersecurity expert). These methods show a measurable improvement based on previous research. Their performance can also be measured after implementation since we can use standard statistical methods to compare their risk assessments to observed reality. (p.144)

Jack Jones has worked in the information security field for over three decades and has a decade of experience as a CISO. He is also the creator of the Factor Analysis of Information Risk (FAIR) framework. (p.144)

That said, quantification does require critical thinking, which can be a challenge because many people in our profession haven’t done a lot of critical thinking over the years when it comes to risk measurement. (p.145)

The everyday meanings of most terms contain ambiguities significant enough to render them inadequate for careful decision analysis. —Ron Howard, Father of Decision Analysis (p.149)

Once you have defined what your rows in the table represent, then your next question is how detailed you want the decomposition in each row to be. In each decomposition, you should try to leverage things you know—we can call them observables. (p.152)

They also realized that many of the challenges in real decisions were not purely mathematical. Indeed, they saw that decision makers often failed to even adequately define what the problem was. As Ron Howard put it, we need to “transform confusion into clarity of thought and action.” (p.155)

The difference between this and an informative decomposition is whether or not you are describing the problem in terms of quantities you are more familiar with than the original problem. (p.157)

Decomposition Rule #1: Decompositions should leverage what you are better at estimating or data you can obtain (i.e., don’t decompose into quantities that are even more speculative than the first). (p.159)

Decomposition Rule #2: Check your decomposition against a directly estimated range with a simulation, as we just did in the outage example. You might decide to toss the decomposition if it produces results you think are absurd, or you might decide your original range is the one that needs updating. (p.159)

For example, suppose you need to multiply A and B to get C. Suppose that when A is large, B is small, and when B is large, A is small. If we estimate separate, independent ranges for A and B, the range for the product C can be greatly overstated. This might be the case for the duration and cost per hour of an outage. That is, the more critical a system, the faster you would work to get them back online. If you decompose these, you should also model the inverse relationship. Otherwise, just provide a single overall range for the cost of the impact instead of decomposing it. (p.160)

What do we see when we see a loss of reputation? … One such article about the 2013 Target data breach was titled “Target Says Data Breach Hurt Sales, Image; Final Toll Isn’t Clear.” (p.161)

In March 2015, another analysis by Trefis also pointed out that overall changes in retail foot traffic at the time and competitive positions explain the changes or lack of changes in Target and Home Depot. Marshall Kuypers, who at the time was working on his management science and engineering PhD at Stanford, focused his study on this issues. He explained that, up to that point in time, all academic research “consistently found little evidence that the two are related” and that “signals dissipate quickly and the statistically significant correlation disappears after roughly three days.” (p.162)

Some research shows that a buy‐and‐hold strategy of stocks involved in a major data breach had returns lower than market returns. (p.164)

Responsiveness: If there is a delay in detection or a delay in reporting or inaction at the board level, there will be a bigger impact. (p.165)

In summary, reputation damage might affect stock price depending on the expectations and trust of the market. If the conditions listed above don’t apply, you are better off using a different method than stock price drops to assess reputation damage. (p.165)

Replacing a lot of upper management responsible for cybersecurity. (It may be scapegoating, but it may be necessary for the purpose of addressing reputation.) (p.166)

Test your decompositions with a simulation and compare them to your original estimate before the decomposition. (p.167)

The most important questions of life are indeed, for the most part, really only problems of probability. —Pierre‐Simon Laplace, Théorie Analytiquedes Probabilités (p.169)

Of course, it is legitimate to ask whether subjective probabilities can be valid. Fortunately, … two findings are clear: (1) Most people are bad at assigning probabilities, but (2) most people can be trained to be very good at it. (p.169)

Yes, the validity of subjective estimates of probability can be and has been objectively measured (ironically, perhaps to some). To deny this is a rejection of scientifically validated facts. A cybersecurity expert can learn how to express their uncertainty with a subjective—but quantitative—expression of uncertainty. (p.169)

You may determine that there is a 2% chance of a data breach in the next 12 months large enough to warrant some public announcement. (Note that when putting probabilities on future events, we must always state the period of time or the probability is meaningless.) (p.170)

Calibrated probability assessments were an area of research in decision psychology in the 1970s and 1980s and up to very recently. (p.171)

This research shows that almost everyone tends to be biased either toward “overconfidence” or “underconfidence” about their estimates, the vast majority being overconfident. (p.171)

They also made some disturbing discoveries about how bad physicians are at putting odds on unknowns such as the chance that a tumor is malignant or that a chest pain is a heart attack. (p.172)

In short, researchers discovered that assessing uncertainty is a general skill that can be taught with a measurable improvement. (p.172)

…there are competing philosophies over the definition, and both sides of the debate include many of the greatest minds in math, statistics, and science. (p.172)

Research indicates that even just pretending to bet money significantly improves a person’s ability to assess odds. In fact, actually betting money turns out to be only slightly better than pretending to bet. (p.176)

People who are very good at assessing their uncertainty (i.e., they are right 80% of the time they say they are 80% confident, etc.) are called “calibrated.” (p.176)

But the lack of having an exact number is not the same as knowing nothing. (p.182)

An assumption is a statement we treat as true for the sake of argument, regardless of whether it is true. Assumptions are necessary if you have to use deterministic accounting methods with exact points as values. (p.182)

Hubbard usually gives his training to experienced managers and analysts, most of whom knew they would be called on to make real‐world estimates with their new skills. Dale Roenigk of the University of North Carolina–Chapel Hill gave this same training to his students and noticed a much lower rate of calibration (although still a significant improvement). Unlike managers, students are rarely asked for estimates; this may have been a factor in their performance. (p.189)

Judgments are better and more consistent if they are based on just a few reality checks. For example, if 100 identified risks each have a 5% to 20% chance of occurrence per year, then we are expecting to see several such events per year. If none of those events have happened in the past five years, then we might want to rethink our estimates. (p.192)

He found that if teams consisted of “belief updaters,” that is, individuals who demonstrate willingness to update beliefs based on new information, then teams could collaborate in groups in ways that at least outperformed other teams of collaborators. (p.193)

Some other research Philip Tetlock conducted showed that if two SMEs [Subject Matter Expert] agree, and they happen to both agree that something is more likely than usual, then the event is more likely than the mere average. (p.193)

We are now in possession of proven theorems and masses of worked‐out numerical examples. As a result, the superiority of Bayesian methods is now a thoroughly demonstrated fact in a hundred different areas. —E. T. Jaynes, Probability Theory: The Logic of Science (p.195)

The claim has often been correctly made that Einstein’s equation E = mc2 is of supreme importance because it underlies so much of physics… . I would claim that Bayes equation, or rule, is equally important because it describes how we ought to react to the acquisition of new information. —Dennis V. Lindley (p.196)

It often surprises managers and technicians that we can make any progress at all with just a few data points. However, Bayesian methods demonstrate that sometimes we can just incrementally change our uncertainty even with a single observation. How a single observation reduces uncertainty is obvious in some extreme examples—such as the chance that bypassing a particular control is even possible is informed by observing just one such event. (p.196)

Now, keep in mind that after you implement MFA, more data is gathered with each year of observations. If you went another year and none of the 12 firms observed a data breach yet, then you have 12 more company‐years of data for a total of 24 company‐years. (p.202)

If you need another example to make this concrete, let’s consider one Hubbard uses in How to Measure Anything. Imagine you have an urn filled with red and green marbles. Let’s say you think the proportion of red marbles could be anywhere between 0% and 100% (this is your prior). To estimate the population proportion, you sample 6 marbles, one of which was red. We would estimate the result as we did in the security procedure example above—the range would be 5.3% to 52%. The reason this range is wide is because we know we could have gotten 1 red out of 6 marbles from many possible population proportions. We could have gotten that result if just, say, 7% were red and we could have gotten that result if half were red. (p.209)

We believe this may be a major missing component of risk analysis in cybersecurity. We can realistically only treat past observations as a sample of possibilities and, therefore, we have to allow for the chance that we were just lucky in the past. (p.212)

Although breaches are unpredictable events, the simulation gave us invaluable insight into the risks we could potentially encounter and the intelligence to help mitigate those risks. —Bo Holland, Founder & CEO of AllClear ID (p.213)

Again, if someone tells you this or anything else we discuss in this book isn’t pragmatic, be aware that we’ve had an opportunity to apply everything we’ve discussed so far and many other methods in many situations, including several in cybersecurity. When someone says this isn’t pragmatic, they often just mean they don’t know how to do it. (p.217)

Brunswik was more interested in describing the decisions they actually made. He said of decision psychologists: “We should be less like geologists and more like cartographers.” In other words, they should simply map what can be observed externally and not be concerned with what he considered hidden internal processes. (p.220)

Both Hubbard and Seiersen have now had many opportunities to use this in cybersecurity. In each case, the model was at least as good as human experts, and in almost all cases, it was a significant improvement. (p.220)

…conditional probabilities can be computed based on historical data. (p.229)

The product of the chance of being wrong and the cost of being wrong is called the expected opportunity loss (EOL). In decision analysis, the value of information is just a reduction in EOL. (p.229)

Remember, the beta distribution can only produce a frequency between 0 and 1. Another distribution used in Bayesian methods is the gamma distribution. The gamma distribution can generate any value greater than zero, including values greater than 1. So if we wanted to say that an event has a frequency of 0.6 to 3.5 times per year, a gamma distribution can handle it. (p.235)

We consider a loss that is reported to insurance that is actually paid out or is part of your deductible as material. (p.236)

The Metrics Manifesto: Confronting Security with Data (TMM) was published in 2022. Richard states that this chapter on metrics provided motivation for that book. Condensed material from TMM has been included here to help flesh out concepts from the first edition of HTMA Cyber. Extensive metrics examples in code can be found at www.themetricsmanifesto.com

But once your new investment (people, process, technology) is deployed, you measure it to determine its effectiveness to continually improve its operation. (p.243)

Functional security metrics (FSMs) seek to optimize the effectiveness of key operational security areas. There will be key performance indicators (KPIs) associated with operational coverage, systems configuration, and risk reduction within key domains. There are several security metrics books on the market that target this level of security measurement. One of the earliest was Andrew Jaquith’s Security Metrics (The Metrics Manifesto is a more modern treatment of metrics). Jaquith’s book brought this important topic to the forefront and has a solid focus on what we would call “coverage and configuration” metrics. (p.243)

Mitigation metrics: These are metrics associated with the rate at which risk is added and removed from the organization. An example metric might be “Internet facing, remotely exploitable vulnerabilities must be remediated within one business day, with effective inline monitoring or mitigation established within 1 hour.” (p.244)

Note, code and supporting functions found in this section are located at www.themetricsmanifesto.com . Go to the section title HTMA CyberRisk. Functions particular to this section are in the htmacyber_functions.R file. (p.244)

As events flow in, you are looking to measure the expected rate with which the next event should materialize. In this sense, interarrivals are like canaries in a coal mine. They can help you detect changes in your data‐generating process with a finer grain than arrivals. (p.250)

Are people, process, and technology working together effectively to reduce risk across multiple security domains? (Note: When we say “security system” we typically mean the combination of people, process, and technology.) More specifically, is your system improving or degrading in its ability to reduce risk over time? (p.254)

These “possible breaches” left without follow‐up could mature into full‐blown, long‐term breaches. (p.258)

The approach described above is effective at measuring cybersecurity risk in a way that drives continuous improvement. For my team, it meant aligning HR and IT on the need to improve their processes for timely deprovisioning. (p.262)

What’s dead, or should be, is slow, cumbersome approaches to doing analytics that add no strategic value. (p.264)

…performance of host‐based versus inline defenses against spear phishing. Or, in the same vein, how often did application whitelisting have impact where all other controls failed—that is, is there a class of malware where application whitelisting is the last line of defense? (p.267)

Interestingly, in dimensional modeling, a fact in a fact table is also known as a “measure.” They’re called a measure because they measure a process at the atomic level. “Atomic” means you cannot decompose the event being measured any further. (p.267)

What distinguishes this tome from its predecessors, How to Measure Anything: Finding the Value of “Intangibles” in Business and The Failure of Risk Management, is that this book is domain focused. More than that, it’s designed to be a road map for establishing cybersecurity risk management as the C‐level strategic technology risk practice. (p.275)

The CSRM [Cyber Security Risk Management] function is a C‐level function. It could be the CISO’s function, but we actually put this role as senior to the CISO and reporting directly to the CEO or board. Of course, if the CISO reports to those functions, then this may work, but it requires an identity shift for the CISO. (p.275)

Also, ongoing risk tracking against tolerances is required for completed models. (p.277)

…nothing in the evidence of recent major breaches indicates that the existing methods were actually helping risk management at all. (p.282)

Standards organizations must end the promotion of risk matrices as a best practice and promote evidence‐based (scientific) methods in cybersecurity risk management unless and until there is evidence to the contrary. (p.282)

To supplement the first point, standards organizations must adopt evidence‐based as opposed to testimonial‐based and committee‐based methods of identifying best practices. (p.282)

An organization could be formed to track and measure the performance of risk assessment methods themselves. This could be something modeled after the NIST National Vulnerability Database—perhaps even part of that organization. (We would argue, after all, that the poor state of risk assessment methods certainly counts as a national vulnerability.) Then standards organizations could adopt methods based on informed, evidence‐based analysis of alternative methods. (p.282)

Certification programs that have been teaching risk matrices and ordinal scales must pivot to teaching both proper evidence‐based methods and new methods as evidence is gathered (from published research and the efforts just mentioned). (p.282)

Auditors in organizations must begin applying the standards of model validity equally to all methods in order to avoid discouraging some of the best methods. (p.283)

Regulators must help lead the way. We understand that conservative regulators are necessarily slower to move, but they should at least start the process of recognizing the inadequacies of methods they currently consider “compliant” and encouraging better methods. (p.283)

Vendors, consultants, and insurance companies should seize the business opportunities related to methods identified in this book. The survey mentioned in Chapter 5 indicated a high level of acceptance of quantitative methods among cybersecurity practitioners. The evidence against some of the most popular methods and for more rigorous evidence‐based methods grows. Early promoters of methods that can show a measurable improvement will have an advantage. Insurance companies are already beginning to discover that the evidence‐based methods are a good bet. (p.283)

Sometimes, Hubbard Decision Research is asked to completely replace ERM where part of that effort included cybersecurity. There are other cases where the initial objective was only to develop quantitative methods for cybersecurity, but after seeing the improved methods, management becomes interested in applying that approach to all risks. (p.284)