May 27, 2014

Creating Real Security Awareness: Step 7 Evaluate


The Design Process * Evaluation
The Design Process * Evaluation
(Photo credit: Jordanhill School D&T Dept)
Welcome to our Creating Real Security Awareness series' eighth installment; step seven, evaluate. Once people get past the execution phase of an awareness campaign, they assume they're done. Unfortunately this means they miss out on a beneficial step: the evaluation step. The evaluation results help to improve programs and justify future awareness program ventures. Since this step can be tricky for some to wrap their head around, many simply overlook it. 

To evaluate, you need to have meaningful metrics, and that will vary upon your objective. Converting an awareness campaign into measurable metrics is where many get stuck, because they failed to properly set themselves up for success.

How do you measure awareness?

Measurements are broken down into two types: measurement of performance (M.O.P.) and measurement of effectiveness (M.O.E.). Of the two different types, M.O.P. is the easier one and fortunately the first one we look at. You cannot measure effect if the awareness campaign was not performed properly. Basically, did you (or your supporting team) execute the campaign like you intended? Did the digital sign display when it was supposed to? Did the article get published on schedule? Did you do what you intended to do? What would you do differently? Basically, this is looking at how the campaign was performed.

Measurements of effectiveness is trickier and all depends on your primary campaign objectives. During this part of the evaluation, you're trying to determine the campaign's effectiveness in communicating the intended message. Back in step four, you listed in your communication plan the campaign's primary objective. If your primary objective was something vague as "raise awareness," you will run into difficulties in trying to measure it.  Your earlier work will help or hinder you during this phase.

As I asked earlier, how do you measure awareness? A reason people skip the evaluation part is largely due to selecting generalized objectives instead of something rather specific. For instance, an objective for an awareness campaign from a previous job was to educate the workforce about the existence of a new internal security site that contained various job aids, frequently asked questions, updated threat information and other resources. The objective is specific. Prior to the campaign, the site received little traffic; however, traffic dramatically increased during and after the campaign. The site's traffic was a M.O.E. that helped me to determine the campaign was a success.

What about M.O.E.s for objectives that don't naturally have built in statistics?
Let's take a look at a couple of the objective examples from the Create Communication Plan part of this series.

- Increase awareness of reporting requirements
- Increase workforce compliance

These don't have my easy website traffic measurement, but there are ways to measure effectiveness. Have security infractions/violation reporting decreased or increased since starting the campaign? Reporting records could be your measurement. Random spot checks conducted before, immediately after, and 60 days after the campaign could provide supportive evaluation measurements. Do you have to provide less security reminders?

A simple survey gathering feedback from your targeted audience could give you data to determine effectiveness by gauging how much of the information members from the workforce remember. When developing surveys, be respectful of the responder's time by keeping the survey short 3-5 questions (no more than 10 max). When possible try using a rating scale (on a scale of 1 to 5) or multiple choice. Questions requiring lengthy write-in answers are less likely to be answered. Free online survey services like SurveyMonkey and FluidSurvey can help facilitate anonymous survey results and easily sort the information for you.

After completing the evaluation, you capture lessons learned and repeat the cycle over again. If you kept your previous notes, the next cycle iteration should go smoother.

No comments:

Post a Comment