Security ratings services tend to be loved or loathed. Loved if you consume them and it makes your job easier, especially if you have no other method of assessing the security of organizations that you need to review. Loathed if you’re on the receiving end and have to continuously respond to questions on your rating and deal with the potential inaccuracies. Or, perhaps, you live with them and get some marginal benefit but mostly just tolerate their existence.
I take the view that we should not only live with these, but work together to improve them because the wider use of even marginally accurate ratings will be a net positive over time. So, let’s start to unpack this in what I hope will be an even handed discussion, and to be clear, I’m not going to talk about any specific ratings service or company.
Are Ratings Necessary?
Let’s try a thought experiment. Imagine there was a ratings service that actually was accurate, fair, updated in real-time and sufficiently well modeled that a good rating corresponded to good security - as best as we can define that. Would you want to use it to rate your vendors or any other entity for which you need need to oversee? I think the answer to that would be, of course, yes. You’d probably also want to run it against yourself constantly.
Now, the question is whether this is ever achievable. If the goal is perfection or nothing, then it is not achievable because of constraints on visibility, timeliness and, more fundamentally, our continued inability to precisely and consistently define what “good” is as it relates to security. So, the next question is whether it can be approximated to be more useful than not.
Ratings exist in pretty much all other industries and professions, from safety, medical, to finance. I don’t think any rating approach in any of these is wholly accurate or has ever claimed to be so. But, pretty much all are useful within certain contexts. For example, a person’s credit score is rarely a wholly accurate representation of their ability to consume and repay credit but it’s useful enough within a margin of error to permit a gigantic market to mostly function reasonably well.
In most of these spaces there is tremendous partnership between the raters and those being rated, often organizations have whole teams to help manage the ratings process to appropriately challenge it and make sure it is indeed fair and accurate. In many cases there is independent governance, or an ombudsman process, to oversee this correct operation. There is strong governance because the consequences of ratings inaccuracy are high, and if it does fail there are significant adjustments - often publicly visible.
Ratings are necessary to bring transparency to risks to enable us to work with our suppliers, counter-parties, and customers to identify and resolve security risk. They will never be perfect but I think we can all partner to keep improving them such that they are more useful. Arguably, if you’re an insurance company, an investor, a non-expert customer, a regulator or other constituent then ratings are already a tremendously useful signal not an absolute measure.
Answer: it is entirely feasible for a ratings approach and service to be more useful than not. Yes, perhaps a low bar, but this will be progress.
Are They Accurate?
Answers to this question tend to be quite polarized. In my experience ratings are not accurate enough to replace the need to do your own supplier risk assessments informed by more in depth internal assessments, either directly or through some other mechanism like a SOC2 (or other accredited testing method with the right scope). Although, if we want to go there, many accreditation and certification schemes can suffer the scope, visibility and timeliness challenges of ratings as well.
However, I think most ratings I’ve seen and used are accurate enough to be used in the negative. In other words, a good rating may not reliably tell you an organization has good security. But generally speaking a bad rating does give you a strong signal that there is something wrong. Similarly, significant volatility in a rating gives you another signal that something isn’t right. I’ve seen multiple situations where large suppliers, who are good at passing conventional assessments have very low or volatile ratings that we successfully used as a trigger to dig deeper and find some issues that needed to be dealt with.
The models of many ratings services suffer from inaccuracy due to the means by which they identify the organization’s assets in scope or because they do not get direct internal visibility to an organization to actually measure more of the things that need measuring. To be fair to the ratings companies, a lot of organizations don’t actually do a great job of keeping their visible perimeters in good order so that even they know this accurately either. Even fewer would permit ratings companies to position sensors, other equipment or data collectors to form the internal view. But, that will change and I know at least one emerging ratings service has a very credible and well thought through approach to achieving this.
Answer: not enough currently but they are on a path to improvement. However, they are mostly accurate enough to be used as a negative signal or as a useful potential contra-input to other assessment approaches.
Is the Industry “Sketchy”?
As with any product or service area there are differences in companies and how they manage the perception of their service. Often the perception of sketchiness is down to certain over-zealous sales and marketing practices. Like with other sector’s or domain’s ratings this can all be resolved with good governance and oversight.
The need for such oversight in security ratings was one of the reasons I first drafted the Principles for Fair and Accurate Security Ratings which were further refined by a number of my colleagues in other Banks and then published by the US Chamber of Commerce (extract below).
Answer: generally no, but often spoiled by overly zealous marketing claims and tactics which we should counter with the Principles for Fair and Accurate Security Ratings.
How are Ratings Used?
I’ve seen ratings used mainly in the following contexts:
Insurance companies. Typically to inform the underwriting process for cybersecurity and related insurance policies. I’ve yet to see (but I might be wrong) ratings used opaquely to feed pricing engines that are beyond challenge in the coverage purchasing / adjustment process.
Investors. Some public and private market investors use ratings as a signal, often a negative not positive signal, as to the cybersecurity health of the companies in their portfolio. The most frequent use case is be alerted to significant negative change to prompt either a "Help!" or "Help?" response from the portfolio risk team.
Regulators. Some regulators in some markets use ratings as part of data gathering prior to examinations to inform and provide material to discuss during the examination.
Customers. Most of your customers will have little capability to assess you to the depth they might need and may have to resort to using ratings.
Boards. I’ve seen early indications from some companies that their Boards, directly or via the Risk or Audit teams are consuming ratings as means to challenge management, including the CISO.
These all need careful and professional handling, even if you believe the ratings they are using are not an accurate representation of your true security posture. Specifically:
Manage your ratings (see section below) so there’s never a surprise on how you are rated. If you are concerned about the cost of consuming your own ratings then appeal to the Principles for Fair and Accurate Security Ratings that any rated company should be able to get their detailed report from any rating company at any time - free.
Be prepared to explain your rating to those using it when they query you. Angrily belittling the rating service or their approach to the person that has likely spent a bunch of money on that rating is not going to make the conversation go better. Patient explanation and suggestions for alternate means of assessing you in a way that make that persons day go a little bit smoother are more likely to pay off. Over time, they will eliminate the rating services that are least accurate because that causes them the most noise and work in using them.
Answer: they are used where information asymmetries exist that need to be resolved where there is no scalable risk measurement alternative, in other words, it’s better than nothing.
How will they Evolve?
Whether you love or loathe security ratings I believe they are here to stay. As above, they are here to stay for the simple reason there is a clear market demand to fill the information asymmetry between rater and the rated. This is needed to permit other business goals to function (investment, insurance, governance, supply chain risk management). The only question is in what form will this take and how quickly it will evolve.
The evolution is inevitably going to have to shift to in-depth internal assessment either through sensor placement, ingestion of existing sensory data, to systemized inspection of other audit and certification reports - or a combination of all of this. There will be a need for more transparent back-testing of realized risk, for example, incident occurrence vs. the predicted risk indicated by the ratings. Many organizations will resist all of this until there are clearer incentives to do so, such as:
Insurance premium reduction.
Contractual requirements.
Investment conditions (pre- and post-investment).
Embedding of security ratings into other, more commercially impactful ratings, like your organization's credit rating. Imagine security being a factor in a credit rating downgrade and what that would do to your view on managing your rating.
Board directives.
Most of these will likely be irresistible.
Answer: security ratings (whether they are ready for it or not) will be embedded in other commercial contexts, this will accelerate the pressure to increase accuracy and open up companies to more direct inspection.
Should You Manage Your Rating?
For ratings of any kind there is always a level of angst about rating accuracy. Most organizations that are rated on anything devote time and effort to manage these ratings. They also establish industry-wide governance to shape the production and use of ratings.
Devoting some small effort in your organization to manage your ratings is worthwhile vs. being constantly victimized by them. I’ve seen it work best either in the vulnerability management team or sometimes in the supplier risk team (taking the self-view of how your own organization is viewed as a supplier). If you still don’t like the idea of spending effort on this then that watch out for that being a false economy. For example, let’s say one of the ratings services has given you an inaccurate low rating. Are you going to spend 2-3 hours to challenge and correct that or are you going going to spend, say, 50 x 30 minutes (25 hours) explaining it to 50 or more customers or other consumers of your rating.
If you are in any medium to large organization your company will be full of people managing ratings (product, safety, ESG, hiring appeal, brand, credit, earnings guidance, and so on) and so seeking some small budget to build an approach to manage your ratings might not seem at all strange. In fact, somewhat counter-intuitively (and perhaps annoyingly) it might even be seen by your executive leadership as a sign of security being a more mature discipline just like all the other things they deem important that are already rated (never mind how imperfectly).
Answer: yes, even if you’re in the loathe camp it can often make simple economic sense to make sure your rating is as accurate as it can be - or at least not inaccurate.
Bottom line: security ratings as they stand today are not what we need them to be. Some aspects of existing approaches are useful, like negative signals, others are not. But we need security ratings, and even if you personally don’t think we do then remember there is enough wider market demand that they will continue to exist and improve. The question is do you want to be part of the process of improving it? The more each company being rated and using ratings hold to the US Chamber of Commerce Principles for Fair and Accurate Security Ratings and is open to working on new approaches to solve this problem the better off we will all be.