Ground Truth Competency Assessment for Smart Grid Cyber Security - - PowerPoint PPT Presentation

Ground Truth Competency Assessment for Smart Grid Cyber Security

TCIPG May 4, 2012 Michael Assante President, NBISE David H. Tobey, Ph.D. Director of Research

The Need for Cyber Defenders

2

"Everywhere I go, across the country, CEOs and business leaders tell me that

• ne of their chief concerns is having the highly skilled workers they need to

power their companies. They believe, and this administration believes, that a globally competitive economy requires a globally competitive workforce…” Secretary of Commerce John Bryson

Grid Modernization

• Explosion of interconnected automation & intelligence
• Growing complexity & sophistication of cyber concerns
• Develop adaptive & resilient systems (people & technology)
• Onboard new & transforming our current workforce

21st Century Grid – Cross Cutting Challenge of Cyber Security

Demand Conventional & Hydro Generation Demand Response Nuclear Energy Efficiency

Plug-In Hybrid Electric Vehicles / Storage

Rooftop Solar / Local Wind Development Wind & Variable Generation

“smart grid”

Drivers Policy Security Economic

cyber security reliability

Building the 21st century grid requires a comprehensive and coordinated approach to policy and resource development – looking at the grid as a whole, not as component parts.

Growing Numbers/Complexity

1960 1970 1980 1990 2000 2010 Devices

Mainframe Mini PC Internet Hosts Devices & Microcontrollers

Organizations Departments People Devices Things

Requires major productivity gains and accelerated skill development

NBISE’s Mission

Our mission is to increase the security of our nation’s critical infrastructure by improving the science by which we identify and measure the proficiency, the performance, and the potential of the cyber security workforce. Our vision is to establish a national, open source library of assessment, instruction, target practice, and performance support tools that advance cyber defense skills through combining lessons learned from cognitive science and psychometrics. We seek to codify, validate, and disseminate evidence-based practices for accelerating the development of competencies that enable effective performance in addressing the growing cybersecurity threat.

How can we assess and develop the future cybersecurity workforce?

• 6. Ongoing

Performance Support & Simulation

• 1. Job

Definition and Competency Analysis

• 2. Aptitude

Assessment 3. Instructional & Simulation Design 4. Proficiency and Performance Assessment 5. Professional Development Plans

Source: Assante, M. J. & Tobey, D. H. (2011) Enhancing the Cybersecurity Workforce. IEEE IT Professional, 13: 12-15.

Moving from summative to formative assessments

Example from NBME Examinee Performance Profile

Foundational Support to Achieving the Benefits of Grid Modernization

• Purpose:

The project contributes to the Department of Energy’s efforts to develop a competency model and explore assessment methods focused on the job responsibilities and unique skill set of Smart Grid cybersecurity specialists. This work is designed to provide a foundation for industry’s ongoing efforts to transform and develop the workforce necessary to achieve the benefits of grid modernization.

• Who:

Those primarily responsible for operational security functions for day- to-day operations, but not engineering and architecture, in smart grid environments.

• How:

Examination of the technical, problem solving, social and analytical skills used by senior cyber security staff in the daily execution of their responsibilities.

• Verify:

A measurement model for assessing knowledge, skills, and abilities in the areas of technical and operational skills.

Subject Matter Expert Panel and Advisory Group

Panel Officers

• Chair – Justin Searle UtiliSec
• Vice Chair - Scott King Sempra Energy

Advisory Group

• John Allen – IEIA Forum
• Joel Garmon - Wake Forest Baptist Medical Center

(CISO)

• Dr. Emannuel - Hooper Global Info Intel and Harvard
• Bill Hunteman
• Jamey Sample - PG&E

Panel Members

• Lee Aber - OPower
• Sandeep Agrawal - Neilsoft Limited
• Bora Akyol - PNNL
• Andres Andreu - NeuroFuzz, LLC
• Balusamy Arumugam - Infosys
• Chris Blask - AlienVault
• Andy Bochman - IBM
• Jason Christopher - FERC
• Art Conklin - University of Houston
• Benjamin Damm - Silver Springs Network
• Anthony David Scott - Accenture
• Steve Dougherty - IBM Global Technology Services
• Ido Dubrawsky - Itron
• Michael Echols - Salt River Project
• Dr. Barbara Endicott-Popovsky - University of Washington
• Cliff Eyre - PNNL
• Maria Hayden - Pentagon
• Charles Reilly – Southern California Edison
• Craig Rosen - PG&E
• Scott Saunders - SMUD
• Chris Sawall - Ameren
• Paul Skare - PNNL
• Clay Storey - Avista
• Dan Thanos - GE Digital Energy
• Kevin Tydings - SAIC
• Don Weber - InGuardians
• Mike Wenstrom - Mike Wenstrom Development Partners
• Nic Ziccardi - Network & Security Technologies

Panel Member Representation

Service Gov't Industry Research Vendor

Security Testing & Smart Grid

Source: Searle, Justin (2012) AMI Penetration Test Plan, National Electric Sector Cybersecurity Organization.

Current Draft OST Competency Model

Constructs (Goal & Task Categories) Constructs (Goal & Task Categories) Manage the project

• Develop project plan
• Monitor project plan

Mitigate vulnerabilities

• Identify resources
• Plan and document actions
• Communicate actions

Identify the critical vulnerabilities

• Identify critical vulnerabilities
• Analyze and map critical vulnerabilities
• Develop and execute mitigation strategy

Understand and demonstrate impact

• Specify target-specific impact
• Determine implications/plan response
• Communicate impact

Penetrate targets

• Identify targets to penetrate
• Analyze targets to penetrate
• Develop and execute penetration strategy

Educate team and clients

• Educate team
• Educate clients

Exploit vulnerabilities

• Infrastructure
• Web and Applications
• Other
• Perform tasks in a safe and lawful fashion

JOB PERFORMANCE MODELING

Developing the Science of Competency Assessment:

Narrow Broad Knowledge

(understanding of strategy or procedure)

Inconsistent Consistent Shallow Deep

Skills

(consistency of performance)

Ability

(transfer across domains)

= Master = Apprentice = Journeyman = Novice

What is a competency?

Source: Tobey, D. H. et. al. (in press) Predictive Performance Modeling: An innovative approach to defining critical competencies that distinguish levels of performance," National Board of Information Security Examiners, Idaho Falls, ID, OST Working Group Report NBISE-OST-11-01

Defining proficiency at multiple levels for multiple roles

Source: Tobey, D. H. et. al. (2011) Predictive Performance Modeling: An innovative approach to defining critical competencies that distinguish levels of performance," National Board of Information Security Examiners, Idaho Falls, ID, OST Working Group Report NBISE-OST-11-01

Critical Differentiation Analysis

#1 … #2 … #3 … #4 … #5 … #1 … #2 … #3 … #4 … #5 … #1 … #2 … #3 … #4 … #5 … #1 … #2 … #3 … #4 … #5 …

Task Differentiation Task Criticality

Low High Low High

Defining the path to performance

Masters Apprentices Journeymen

Source: Tobey, D. H. et. al. (2011) Predictive Performance Modeling: An innovative approach to defining critical competencies that distinguish levels of performance," National Board of Information Security Examiners, Idaho Falls, ID, OST Working Group Report NBISE-OST-11-01

Self-Assessment Instrument

Knowledge (Understanding)

• Learning Modes/Hours
• Degree of understanding
• Degree of difficulty

Skill (Consistency)

• Self-efficacy scale
• Frequency scale
• Performance scale

Ability

• Planning and Monitoring
• Problem Solving

Performance Levels

• Red: Low performance
• Yellow: Borderline performance
• Green: High performance

2 4 6 8 10

Identify Vulnerabilities Exploit vulnerabilities Mitigate Vulnerabilities Penetrate Targets

Individual Competency Profile: Radial Chart

Views by Composite, Knowledge, Skill, and Ability

Individual Competency Profile: Drill-down (Penetrate targets)

Foundational Tasks

Composite score

Comparative score

Identify ownership of gateway devices (16.77)

83.8 Average

Identify recon that is within project scope (15.63)

46.8 Low

Search online sources for useful information about a target (15.45)

53.5 Average

Differentiating Tasks (with weights)

Analyze data found on compromised machines to enable exploitation deeper into the network (24.02)

36.0 Average

Identify major assets subject to attacks (23.67)

87.2 High

Identify targets for potential exploitation (23.67)

56.0 High

Analyze data found on compromised machines for strategic value as seen by a worst case attacker (23.60)

26.2 Low

Overall Score

My Score

54.9 Average

Knowledge Skill Ability

Identify recon that is within project scope

86.5 27.4 47.3

Workforce Planning

Source: International Center for Leadership in Education (ICLE) Rigor/Relevance Framework (Daggett, 2000)

Rigor/Relevance Framework Analysis

• Symbols for major competencies
• Instances represent task areas
• Location of instances represent

current achievement (KSAs) by quadrant: 1. Foundational Knowledge 2. Foundational Skill 3. Differentiating Skill 4. Ability

• Drill down on any instance

to a personal development plan

Personal Development Plan

State Analysis What are my strengths? What are my weaknesses? Feedback I have received? Conclusion 1__________, 2.__________, 3__________, 4._________, 5.___________ 1__________, 2.__________, 3__________, 4._________, 5.___________ 1__________, 2.__________, 3__________, 4._________, 5.___________ 1__________, 2.__________, 3__________, 4._________, 5.___________ Resource Listing Reference work #1 Reference work #2 Online course #1 Online course #2 US-Cert: Cyber Security Policy Planning and Preparation – PDF download NIST: Guide to Industrial Control Systems (ICS) Security – PDF download SANS: Hacker Techniques, Exploits & Incident Handling – LMS course SANS: Reverse-Engineering Malware: Analysis Tools and Techniques – LMS course Improvement Timeline When do I want or need to achieve the desired state? What is my schedule to work on these focus areas? Focus Area #1 : Short term________; Medium term_______; Long term_________ Focus Area #2 : Short term________; Medium term_______; Long term_________ Focus Area #1 : Day / time 1______; Day / time 2_______; Day / time 3________ Focus Area #2 : Day / time 1______; Day / time 2_______; Day / time 3________ Notes Document any important notes or other items in this section

Clicking on any symbol instance in the RRF brings up the table

The GTED Cybersecurity Performance Management System

• Workforce

Planning and Development

• Organization

Assessment Summary

• Individual

Assessment Summary

• Task

Assessment Results with KSA Metrics

Competency Model Content Model Object Model Development Model

Advanced Defender Aptitude, Performance Testing and Simulation (ADAPTS) Framework

Adapted from Ostyn (2005) Competency Data for Training Automation

Training Objects (CourseLets) SCORM/CMI-5 XML Library Assessment Objects (TestLets) Simulation Objects (SimLets)

• Assessments
• Content
• Metrics
• Simulations
• Etc.

Why job performance models are important

 Facilitate translation of functional roles into job roles  Clearly distinguish knowledge, skill, and ability  Determine factors that differentiate performance at varying levels of skill  Identify the critical factors that predict performance  Competency models describe Job performance models prescribe

26

JPM’s help to determine who should be developed (aptitude), how to development them (skill profiles), and when they are ready to take the next step (performance-based learning)

Questions?

Contact: Michael Assante Michael.assante@nbise.org