What We Learn from Cyber Exercises, or Not Jim Duncan CSIRT - - PowerPoint PPT Presentation

2007 FIRST Annual Technical Conference – Sevilla, España

What We Learn from Cyber Exercises,

• r Not

Jim Duncan CSIRT Coordinator, BB&T 2007 June 20

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

2

Overview

• Background
• Purpose of exercises
• Examples of what we can learn…
• And what we fail to learn (repeatedly)
• Purpose of exercises, redux
• Future improvements
• What else?

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

3

Background

• General cyber-security expertise with a

special focus on incident response

• Product security work as well as critical

infrastructure protection issues (ISACs)

• Varying amounts of involvement with

many different cyber exercises including Cyber Storm, Livewire, various ISACs…

• And many real disasters, too
• Exercise details not included

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

4

Why Conduct Exercises?

• So we know what to expect & what to do!

▪ People in disasters fall into 3 categories:

• 10% -- 15% remain calm & act quickly
• 15% or less COMPLETELY FREAK OUT!
• Remainder are “stunned and bewildered”.

[John Leach in Aviation, Space, and Environmental Medicine, 2004]

▪ Survivors anticipate & plan accordingly ▪ Do you review the safety card every time you fly?

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

5

What Can We Learn?

• How will we react?
• Who will be the real stakeholders?
• What capabilities will succeed or fail?
• What are the unforeseen obstacles?
• What serendipity awaits us?
• What better estimates can we calculate for

cost-benefit analyses?

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

6

How Will We React?

• Perhaps the most obvious goal is to test

an organization’s response to a crisis

• When handling new information, the brain

slows down (e.g.,1977 Tenerife accident)

• Under stress, it slows down even more!

45% of people “shut down” in a crisis

• Minimize “milling”; time is very valuable
• Mitigate “disbelief”; Act now!

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

7

Who Will Be the Real Stakeholders?

• A critical point in the development of an

incident response plan is to identify who has authority over an asset and who pays for it, too; they might not be the same unit

• Exercises have the potential to expose

that information, at times with great relief

• Results should be included in plan review
• Good justification for exercise

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

8

What Capabilities Succeed or Fail?

• Text paging has failed, but not noticed

because the monitoring system pages the

• perators to report problems
• How many of you provision your support

teams with toll-free numbers?

• How many of you know that toll-free

dialing won’t be available in a disaster?

• Or that it can’t be dialed from
• utside the region (overseas)?

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

9

What are the Unforeseen Obstacles?

• Another obvious reason for an exercise;

many hope to find the “gotchas” before a real crisis occurs

• Unfortunately, it’s based totally on luck
• TIP: review your toll-free number uses
• TIP: make sure your teams really know

how to use PGP and have had their keys signed & published

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

10

What Serendipity Awaits Us?

• Exercises are a good thing, and every one

in which I have participated has produced valuable results with practical application

• It’s easy to forget about positive stuff when

we worry so much about negative things

• One example: other teams rewrote my

faux advisory and discovered aspects that hadn’t occurred to me earlier

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

11

What Estimates Can We Calculate?

• Cyber security is catching up with metrics
• Still horribly lacking with incident response
• Exercises can expose unforeseen costs as

well as unanticipated rewards

• Both help to reinforce the value of CSIRTs

to management up to the board room level

• Also helps to reveal intangibles like

sharing opportunities and potential future relationships

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

12

What We Fail To Learn

• We fail to bring in the existing experts
• We fail to discover existing stakeholders,

groups, capabilities, relationships

• We fail to assess authority & responsibility
• We fail to appreciate the resources and

time involved in anticipated responses

• We fail to imagine the threats
• We fail to keep it secure

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

13

We Fail to Bring in the Experts, 1

• FS-ISAC tabletop considered a power

failure at a telephone switching facility due to sabotaged diesel backup systems

• Organizers unaware of battery systems

and alternative fueling systems

• Credibility was suspended and the

participants were unmotivated

• Value of exercise questionable

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

14

We Fail to Bring in the Experts, 2

• Exercise planners spent considerable time
• n scenario involving railroad cars and the

lack of real-time tracking ability; expected major fumbling by participants to resolve

• In reality, locomotives are needed to move

train cars & their locations are well known!

• As before, credibility was suspended, etc.
• Exercise value plummeted!

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

15

We Fail to Discover Current Players

• “FIRST” means many things to many folks

▪ The “Federal Incident Response Support Team” might not be who you think it is; insist on clarification

• Misunderstandings about FIRST influence

incorrect conclusions favoring involvement ▪ Information sharing ▪ Web of Trust

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

16

We Fail to Assess Authority, 1

• ISACs are defined per CIP sector for

information-sharing and analysis ▪ IT-ISAC handles information technology ▪ Telecom-ISAC handles telephony ▪ Who handles the ISPs? Each ISAC says the other has superior authority

• And the ISPs “just want to be

left alone, thank you…”

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

17

We Fail to Assess Authority, 2

• The U.S. National Response Plan divides

activities by defined functional areas

• Emergency Support Function #2 handles

telecom and information technology, while ESF#7 supports office equipment

• When a server in a disaster agency’s

remote field office starts attacking other systems, who will handle it?

• Answer: “No one, immediately”

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

18

We Fail to Anticipate Response Cost

• Most plans (and thus most exercises) are
• riented toward physical events
• In cyber-space, most planning ignores the

international angle (Cyber Storm is trying hard to get this right, and will succeed)

• For example, for an international attack I

was instructed to notify the Department of State’s 24-hour Watch Desk...

• Guess how long that takes!

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

19

We Fail to Imagine Threats

• Following Hurricane Katrina, IT/Telecom

restoration initially followed rules oriented toward public safety, not toward critical infrastructure protection issues

• A major bank couldn’t get essential parts

for back-office transaction processing

• “Instant cash” was unusable because bank

was completely unreachable

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

20

We Fail To Keep It Secure

• Multi-site exercise connected to the

Internet reduces cost but poses risks

• Collected diverse set of security experts

connect to web pages for net simulation

• Traffic is not SSL-enabled nor tunneled
• Links to “bad sites” were genuine and

HTTP referrers had not been disabled!

• To their credit, Cyber Storm

staff fixed that within hours

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

21

Purpose of Exercises, Redux

• Pre-identify essential groups which will

provide better coverage of stakeholders

• Improve rewards for active participation
• Eliminate the “Yet Another Group” problem

(include FIRST; spell it out if necessary)

• Constrain novelty-for-novelty’s-sake-alone
• Identify and preserve your

group’s corporate knowledge

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

22

Future Improvements

• Invite the experts; inform them, trust them
• Research and approach existing groups;

don’t start your own until and unless you are certain a collaboration will not succeed

• Test your equipment and methods as

realistically and thoroughly as possible

• Assume issues are global
• GET INVOLVED!

Duncan - What We Learn from Cyber Exercises, or Not 2007 FIRST Annual Technical Conference – Sevilla, España

23

What Else?

• Contact information:

James N. Duncan, CISSP Ji m . Duncan@

• BBandT. com

j nduncan@ gm ai l . com +1 919 334 4318 (office) +1 919 608 0748 (mobile)

ht t p: / / www. Li nkedI n. com / i n/ Ji m Duncan/

• Questions and Answers?