Calibrated Probability Assessment

0  comments

We recently took a client through a subjective probability assessment calibration training session and upon seeing their results (below), they asked the question “what do these numbers mean?”

Screen Shot 2016-02-15 at 1.47.34 PM
I responded with a brief email  that I thought was worth sharing because it helped them have a better understanding of the “numbers” and their significance (or so they said). Here is what I sent them.


“Kevin, Good question. These numbers essentially mean that the five of you collectively, and for the most part individually, are capable of providing fairly accurate subjective probability assessments (the group showed marked improvement between the first and the last exercises).  

This is important because as we discussed, humans are generally very bad at estimating probabilities, especially when there are multiple variables or factors involved. There are a number of reasons for this shortcoming including tendencies called cognitive biases, which can impact judgement and decision making (sometimes catastrophically).


Being “calibrated” is a valuable skill when performing any type of quantitative analysis, especially risk analysis, as risk by its very definition involves uncertainty and estimating/calculating probabilities. I am sure you have seen folks struggle when trying to describe cyber risk to a business person in qualitative terms (“high/medium/low” or “critical / non-critical”), or in pseudo-quantitative (this risk is a “5” and this one is a “10”). It isn’t very effective and can lead to mis-informed decisionmaking.


Many industries are either looking at or moving toward quantitative analysis. Last year the World Economic Forum actually proclaimed that the world needs to move to do a better job of quantitatively analyzing/measuring cyber risk. Many of these concepts are new to folks, but I have been studying and applying them for about 6 years and have found a lot of good resources including published material, research, and books (both academic and non) that I can point you to if you ever have a case of insomnia. But if you want a couple of short, goodreads, take a look at these two articles:


http://understandinguncertainty.org/node/85


https://en.wikipedia.org/wiki/Calibrated_probability_assessment

{"email":"Email address invalid","url":"Website address invalid","required":"Required field missing"}

You may also like:

Dedicated Vs. Non-Dedicated HIPAA Security Officer
What is a Dedicated and Non-Dedicated HIPAA Security Officer?
What are the primary responsibilities of a HIPAA Security Officer?
What is a HIPAA Security Officer?

Subscribe now to get the latest updates!

>