VIDEO TRANSCRIPT
the very first question that we've got is what is the best way to baseline security awareness levels in an organization what metrics can be used to track behavior improvements and we're going to start with andrew on this one
so andrew over to you
lovely thank you jonathan um the the short answer which i'll get to is um our my favorite metric is the reporting rate of the end user so um the end state of behavioral change is to have your end users if we're talking specifically to phishing attacks um uh actually reporting what they see um so i guess to get to that point um there's a couple of things we need to look at first so um what are the overarching goals of what an organization is trying to do they're all going to be different but i would start off by saying that you need to if you're the admin running a security awareness program to check in with your cso to find out what the organization has been doing and make sure that that's aligned before getting into cyber security a few years ago i spent most of my professional career in and around training in l d and that behavioral change is really difficult and for anyone what you're always asked by management is how are you going to prove that you've changed behavior how are you going to talk about your return on investment or to that you your program has been successful so it's really hard uh that that said there are some metrics that we see used quite a lot to to prove the value of a program and they are around things like the number of malware infections and user machine remediations you see time and resources spent on abuse mailbox management so a reduction in that number of success successful phishing attacks you see from the wild and also downtime hours of users so there are kind of some core things that you can measure but from a from baselining and the successful metrics what i would advise is if you're starting a program um we often overlook the broader sense of risk so some kind of questionnaire to understand people's knowledge of phishing and passwords and physical security and whatever is aligned to the organization so get a baseline from that from a questionnaire which you can repeat down the line the other one which most people do is the susceptibility so send out a phishing simulation so you can understand the percentage of people who are likely to click on a link or enter credentials or open a file so that gives you a baseline before you start with a program to then see how successful you've been but more and more we're seeing because that's susceptibility that failure rate in people clicking on fish has quite a negative connotation the really strong one and what a lot of organizations are moving towards is that reporting rate so once you've told people that you're going to give them a send them phishing simulations if you can give them a reporting button to say i think this is suspicious please take it out of my inbox and send it to someone who can have a look at it um having a increased reporting rate uh repeats so let's say you send out a phishing simulation if you've got 30 50 70 percent of end users properly identifying and saying this is a suspicious looking email i don't know whether it's a simulation or real but i'm going to report it that metric i think is is is really key brilliant okay nigel so look i i think it was a very comprehensive answer from andrew uh so there's not much more to add other than um to agree i think that um intuitively it'd be nice to consider that the number of uh fishing successful fishing tax would be reducing but we all know that um there's multiple campaign shift of the threat environment and you have talent changing in the organization so it's not really possible to track it like that so i do think reporting rate is uh is is really valuable i would just say also that um as you start to get an improvement in the reporting rate i sort of found that even with all the awareness programs that we put together and say report it this way it still gets reported through multiple channels so i think another way of extending it would be to say to start to see that the right channels are being used to report the issues um and that the correct uh reporting into a service desk system is also being used i think that sometimes if um the knowledge is not quite there people will report it as a general issue rather than a security incident and then it takes some time to traverse the system uh but above and beyond i agree the reporting rate is a really good facet brilliant okay trevor you're up next do you have some comments um yeah there's very little to add to what nigel and andrew has said i mean there's lots of metrics to capture um one thing i'd say or add to that is be aware of the culture and the kind of the water cooler test is are people actually talking about security and are they approaching you with that i just wonder style question and that's obviously a good indicator as well because i think it's it's important to have a culture of security rather than a specific metric a lot of people can attend a security awareness session and the they'll take it on board but they'll forget it as soon as they leave but having that culture and those discussions is a good metric to be aware of it's very hard to measure of course but as a security team are you being asked more these kind of questions as well as being asked on the at the home front particularly in the environment we're in people asking more about should i do more with my home wi-fi or i can pick up my neighbor's wi-fi what should i do it's good that you're approachable i think it's very important but um it's probably the only thing i'd add to those two answers brilliant thank you michael okay yes thank you um look wonderful answer from andrea i think it's very hard to to you know to add anything to that just probably a couple of observations when i you know i'm talking to clients about matrix and security awareness is is sort of there's two there's two parts of this there's there's the activity-based reporting which is to say the security awareness has changed the behavior and we see people doing something different versus outcome-based reporting which can be sometimes a little bit harder to quantify you know how much of of the change to our security posture has actually resulted in in this change in activity so this there's an important distinction to make there um but i think generally also sort of explaining what trevor is saying is that there's a whole lot more to security awareness which is more than just the metrics and what i see is sometimes a road blocker with people as they get tied up in this discussion around you know do we need some metrics and and do we need to measure roi on this activity and all these other things and certainly from a business perspective you do but i think there is a whole much more benefit to security awareness training that isn't just largely told by the metrics it's the advantages to your culture knowledge share and some of those other non-tangible aspects as well