Measuring the Effectiveness of Security Using ISO 27001

A white paper by Steve Wright,
Siemens Insight Consulting

Whilst the intentions and objectives behind
ISO/IEC 27001:2005 (ISO 27001) aren’t
dramatically different to those in BS 7799-
2:2002, one of the changes with the
biggest potential impact to organisations is
the requirement to measure the
effectiveness of selected controls - or
groups of controls - within the new
standard (for more details see ISO 27001
Clause 4.2.2 d).

This new requirement not only demands
that businesses specify how these
measurements are to be used to assess
‘control’ effectiveness (there are now 133
Controls in the new standard), but also how
to measure the selected controls’
effectiveness. In addition to this, the new
standard requires that these
measurements are comparable and
reproducible, e.g. so they can be used
time and time again, and compared on a year by year basis to gain a
better understanding on trends, etc.

So why do organisations need to measure security effectiveness in
the first place? Well, it would be easy to say we’re secure, but how
can you demonstrate that an organisation’s security controls are
working effectively, and equally importantly - how can you
demonstrate:
• Ongoing improvement?

• That your organisation has met legal, regulatory and compliance
with standards, contractual requirements such as Sarbanes-Oxley -
SOX, BASEL II, Payment Card Industry Data Security Standard – PCI
DSS, etc.?

• That any future expenditure is based on sound and reasoned
security solutions, (software, hardware, training, etc.) that is
appropriate and effective within your organisation?

• That your organisation is compliant with ISO 27001 (including other
Management Systems such as ISO 9001, Infrastructure Technology
Information Library - ITIL, COSO, CoBIT, ISO 20000)?

• Where implemented controls are not
effective in meeting their primary
objectives - to reduce risk?

• Assurance to auditors, senior
management and stakeholders that risk
justified implemented controls are
working effectively (i.e. they have
invested their money wisely for a good
return on investment - ROI)?

• Where implemented controls are not
effective in meeting their primary
objectives - to reduce risk?

You could be forgiven for thinking this
should be a reasonably straightforward
task. After all, most IT departments
throughout the world have been working
within some kind of measurement
infrastructure (e.g. sometimes measured
within contractual Key Performance
Indicators - KPI, Service Level Agreements -
SLA, Operational Level Agreements - OLA)
since the mid-1990s and should, by rights,
be used to considering how to measure
their IT effectiveness (most IT departments
I’ve known usually have very stringent
methods for providing value for money).

The challenge is, the whole area of what
constitutes good and effective security and
how to implement it is subjective and
therefore difficult to quantify, let alone
provide statistics. In fact, within most
organisations there is often plenty of
evidence that good security practices and
controls are in place, especially those
organisations that have already
implemented management frameworks
such as CoBIT, ITIL or COSO.

So what are the benefits of measuring
your organisation’s security effectiveness?

• Provides tangible evidence of cost
reduction - through better risk
management and reduction of impact
caused by exploitation of threats
• Provides better cost / benefit analysis and
therefore helps ensure ROI decisions
going forward
• Actually eases process of monitoring the
effectiveness of the ISMS (e.g. less labour
intensive, for example, if using tools, and
provides a means of self checking)

• Using proactive tools to measure can
prevent problems arising at a later date
(e.g. network bottlenecks, disk clutter,
development of poor human practices)

• Reduction of incidents and better
understanding of root cause

• Motivates staff when senior
management set targets

• Tangible evidence to auditors and
assurance to senior management that
you are in control - i.e. Corporate
Information Assurance (Corporate
Governance), and top-down approach to
Information Assurance.

Whatever the organisational drivers for measuring
the effectiveness of security, it should no longer be
just about identifying the controls to be
implemented (based on the risk assessment), but
also about how each control will be measured
against its original objective (to reduce the chances
of the risk being exposed). After all, if you can’t
measure it, how do you know it’s working
effectively?

Before deciding which control should be used to
measure effectiveness, your organisation should first
undertake to ensure the following activities:

• Confirm relevance of selected controls through risk
assessment (Mandatory requirement for both ISO
27001 compliance and certification projects)

• Define objectives, ensuring they map back to the
business objectives

• Use existing Indicators wherever possible, e.g. in
ITIL terms, KPIs

• A KPI helps a business define and measure
progress towards a particular goal

• KPIs are quantifiable measurements of the
improvement in performing the activity that is
critical to the success of the business

• Within the Information Security Management
System (ISMS) audit framework, identify controls
which can be continuously monitored, using a
chosen technique

• Establish a baseline, (e.g. security awareness
trained x amount of people in y timeframe) against
which all future measurements can be contrasted /
compared

• Provide periodic reports to appropriate
management forum / ISMS owners (illustrate with
graphs - pictures paint a thousand words)

• Identify Review Input - agreed recommendations,
corrective actions, etc.

• Implement improvements in line with any existing
Management Systems, e.g. ISOs 9001, 14000,
27001, 20000, 18000

• Establish / agree new baseline, review the output,
apply the PDCA approach (i.e. Plan - Do - Check -
Act).

Hopefully, each business may have its own
measurements already in place (e.g. SLAs, OLAs, KPIs).
The challenge is to set a ‘measurement’, which is
realistic based on a previous known figure, and ensure
the future figure is measurable and reproducible.

Senior management and possibly auditors,
are more likely to want to see the big picture,
therefore, consider monitoring the
effectiveness of a group of controls, e.g.
Section 13 of ISO 27001 - Security Incident
Management, plus others. Try to encourage
senior managers to buy into realistic ‘firsttime’
goals, such as measuring how well you
coped with the latest security incident, in
what time frames you held the lessons learnt
meeting and at what point the improvements
to the existing system became operational.

When selecting controls to be measured for
their effectiveness, it is worth bearing in
mind that you could group them into four
categories:

1. Management Controls
Security Policy, IT Policies, Security
Procedures, Business Continuity Plans,
Security Improvement Plans, Business
Objectives, Management Reviews.

2. Business Processes
Risk Assessment & Risk Treatment
Management Process, Human Resource
Process, SOA Selection Process, Media
Handling Process.

3. Operational Controls
Operational Procedures, Change Control,
Problem Management, Capacity
Management, Release Management,
Back-up, Secure Disposal, Equipment
Off-Site.

4. Technical Controls
Patch Management, Anti-Virus Controls,
IDS, Firewall, Content Filtering.

In a changing environment, new baselines
will need to be set each time a major
change or incident occurs within the ISMS,
so this is just the beginning. Try to
establish regular review cycles of your
security effectiveness measurements and
consider how this might improve, how
your organisation can become more
effective in the management of its
incidents.

that security and risk management is a
black art and that it is un-measurable. In
fact, we should start to see tangible
benefits from measuring and improving
our ISMSs and Security Management
solutions globally. For more information,
the new ISO 27004 will soon be published,
which will help those organisations who
are unclear about this and the new
standard should help any organisation get
a grip on measuring the effectiveness of
security.

Author’s biography
Steve Wright is the Senior Consultant
providing professional advice in
relation to Information Security/
Technology/ Management to meet
BS 7799, ISO 27001, ITIL, ISO 20000,
PAS 56, PAS 99, PCI DSS, ISO 13335
and works within legal and regulatory
frameworks such as Basel II, SOX and
Combined Code requirements. In
addition, he is accustomed to
implementing risk best practices such
as Enterprise Risk Management
frameworks and conducting risk
assessments, using tools such as
CRAMM.
Steve is currently project managing
many implementations of ISO 27001
ISMS systems, both virtually and
physically, from initiation through to
final delivery, to meet certification
requirements of ISO 27001, ISO 9001
and ISO 20000 in both financial,
private and public service sectors,
three of which have recently achieved
certification to ISO/IEC 27001:2005 in
late 2006 alone.
Steve heads up the Security
Management service line within
Siemens Insight Consulting managing
the team of six highly qualified and
experienced consultants.

Insight Consulting is the specialist Security,
Compliance, Continuity and Identity Management
unit of Siemens Enterprise Communications
Limited and offers a complete, end-to-end portfolio
encompassing:
• Security • Compliance
• Continuity • Identity Management
• Managed Services • Training
Siemens Insight Consulting subscribes to the CESG
Listed Advisor Scheme (CLAS) and CHECK services.
We’re also certified against ISO 27001 and are a
preferred supplier of services to the UK
Government and an accredited Catalist supplier.
If you’d like to find out more about how we can
help you manage risk in your organisation, visit our
web site at www.siemens.co.uk/insight
Siemens Insight Consulting
Tel: +44 (0)1932 241000
Fax: +44 (0)1932 236868
www.siemens.co.uk/insight