Data-driven techniques for cyber control effectiveness

There are many criticisms online of qualitative cyber control effectiveness measurement techniques like how different assessors can produce different results, how there is inconsistency even from the assessor, how mood plays a factor in assessment, what you had for breakfast, or your educational background. Many people are quick to point out these issues but don't often provide realistic solutions for how to solve this.

In my talk I will ensure that everyone walks away with at least one technique they can use to consistently measure control effectiveness for your most critical cyber security controls. And I don't cherry pick easy controls like "Check if MFA is implemented on all identities" to illustrate my measurement techniques. I will provide example measurement techniques for controls that have been traditionally harder to measure like security awareness training, patch management, and network segmentation. I will demonstrate how statistical random sampling can quantify effectiveness with surprising accuracy using smaller sample sizes than you might expect.

All of the measurement techniques I propose will result in scores scaled from 0-100 represented as percentages. This means that your scores can be used in the future for more quantitative risk measurements. For example: Risk = cost of risk * likelihood of breach * control effectiveness

The first quarter of my talk will set the scene and explain when these techniques are necessary and when they are overkill. Choosing the right tool for the right job is critical to make sure you use your resources well. These approaches don't necessarily save time initially, but they provide consistency and repeatability that makes them worthwhile, especially as they can be executed by junior GRC analysts once designed.

The remainder of my talk will feature example measurement techniques, some theory behind how they work, and some demos of how in the long run these techniques can be automated using tools like Excel, Power BI, and even AI assistance.

This talk is ideal for organisations with at least one dedicated GRC role. For smaller teams, I'll show how implementing this for just 2-4 critical controls can provide actionable data with only about two weeks of effort per year. Larger organisations can scale these approaches to build compelling, data-driven business cases.

This talk comes from significant investigation and development of better measurement techniques, going beyond the surface-level guidance offered by frameworks like NIST 800-55 and ISO 27004 to address those challenging controls they often overlook.

I have spoken at AISA Melbourne and more recently AISA Canberra on similar topics. This AISA Melbourne 2025 talk has been completely updated to include multiple new techniques, practical examples, and automation pathways for continuous control assurance.

After this talk GRC practitioners will be able to measure the effectiveness of any cyber security control using the data that they already have and communicate their calculations accurately. These measurements are designed to go directly into your quarterly cyber security reporting presentations and have a direct correlation to the effectiveness of the cyber security program.