Businesses use predictive metrics all the time. For example, forecasting next quarter’s revenue is a predictive metric used widely in business. When looking at any predictive business metric there are two values that define the usefulness of the metric – its accuracy and precision. Accuracy describes the degree of closeness of measurements of a metric to the measured quantity’s true value. Intuitively this is a measurement of how useful a metric is for predicting an outcome. A business prediction metric that isn’t accurate doesn’t really reflect the quantity’s true value – so its usefulness is limited.
Precision on the other hand describes the degree to which repeated measurements under unchanged conditions show the same results. Precision is often confused with the significant figures of the metric i.e. the “number of decimal points” of the prediction. A metric can be very precise, but also very inaccurate – just take a look at the predictions for the 2020 election. Polls gave Biden an advantage in Florida of 2.5%, but the end the results were 3.3% in favor of Trump. The prediction 2.5% was very precise but completely inaccurate.
Pundits and professional prognosticators use this subtle difference to great effect. By using predictive numbers that are precise (e.g. 2.5% instead of “about 3”), people can be fooled into thinking the prediction is accurate – when in reality it is only precise.
What does any of this have to do with cyber security? Well, this week Microsoft announced that with the launch of the new version of the Security Update Guide, it will describe vulnerabilities using Common Vulnerability Scoring System (CVSS). They should be applauded since any move towards standardization in cyber security is valuable. Since CVSS attempts to predict the risk of a CVE we should look at it as just another business prediction. As a metric, CVSS is very precise. The main question is whether it is also accurate.
The answer is complex. If you are trying to understand the impact of a potential CVE relative to anther CVE without having any information on the organizational context of the vulnerability – CVSS can be quite good. The problem is that organizational context can drastically change the impact and prioritization of a vulnerability. Vulnerabilities are not isolated, they must be analyzed in an organizational context that takes into account data flows, process flows, configurations and access. A single, standardized number cannot predict actual risk from a CVE for a specific organization with any accuracy – no matter how precise the CVSS. To increase accuracy, you need to add organizational context to the CVSS equation.
Orchestra’s Harmony Purple uses its patented Attack Path Scenarios as the context to create much better vulnerability prioritization tailored the actual state and topology of the organization’s network.