Ground Truth Competency Assessment for Smart Grid Cyber Security - PowerPoint PPT Presentation

Ground Truth Competency Assessment for Smart Grid Cyber Security TCIPG May 4, 2012 Michael Assante President, NBISE David H. Tobey, Ph.D. Director of Research

The Need for Cyber Defenders "Everywhere I go, across the country, CEOs and business leaders tell me that one of their chief concerns is having the highly skilled workers they need to power their companies. They believe, and this administration believes, that a globally competitive economy requires a globally competitive workforce…” Secretary of Commerce John Bryson 2

Grid Modernization  Explosion of interconnected automation & intelligence  Growing complexity & sophistication of cyber concerns  Develop adaptive & resilient systems (people & technology)  Onboard new & transforming our current workforce

21 st Century Grid – Cross Cutting Challenge of Cyber Security Plug-In Hybrid Electric Vehicles / Storage reliability Wind & Variable Generation Demand Response Conventional & “smart grid” Demand Hydro Generation Energy Efficiency Nuclear Rooftop Solar / Local cyber security Wind Development Drivers Policy Security Economic Building the 21 st century grid requires a comprehensive and coordinated approach to policy and resource development – looking at the grid as a whole, not as component parts.

Growing Numbers/Complexity Requires major productivity Things gains and accelerated skill development Devices & Devices Microcontrollers Internet People Hosts Devices Departments PC Organizations Mini Mainframe 1960 1970 1980 1990 2000 2010

NBISE’s Mission Our mission is to increase the security of our nation’s critical infrastructure by improving the science by which we identify and measure the proficiency, the performance, and the potential of the cyber security workforce. Our vision is to establish a national, open source library of assessment, instruction, target practice, and performance support tools that advance cyber defense skills through combining lessons learned from cognitive science and psychometrics. We seek to codify, validate, and disseminate evidence-based practices for accelerating the development of competencies that enable effective performance in addressing the growing cybersecurity threat.

How can we assess and develop the future cybersecurity workforce? 1. Job Definition and Competency Analysis 5. 2. Aptitude Professional Assessment Development 6. Ongoing Plans Performance Support & Simulation Source: Assante, M. J. & Tobey, D. H. (2011) Enhancing the Cybersecurity Workforce. IEEE IT Professional, 13: 12-15. 4. 3. Proficiency Instructional and & Simulation Performance Design Assessment

Moving from summative to formative assessments Example from NBME Examinee Performance Profile

Foundational Support to Achieving the Benefits of Grid Modernization  Purpose: The project contributes to the Department of Energy’s efforts to develop a competency model and explore assessment methods focused on the job responsibilities and unique skill set of Smart Grid cybersecurity specialists. This work is designed to provide a foundation for industry’s ongoing efforts to transform and develop the workforce necessary to achieve the benefits of grid modernization.  Who: Those primarily responsible for operational security functions for day- to-day operations, but not engineering and architecture, in smart grid environments.  How: Examination of the technical, problem solving, social and analytical skills used by senior cyber security staff in the daily execution of their responsibilities.  Verify: A measurement model for assessing knowledge, skills, and abilities in the areas of technical and operational skills.

Subject Matter Expert Panel and Advisory Group Panel Members Panel Officers • • Lee Aber - OPower Chair – Justin Searle UtiliSec • • Sandeep Agrawal - Neilsoft Limited Vice Chair - Scott King Sempra Energy • Bora Akyol - PNNL • Andres Andreu - NeuroFuzz, LLC • Balusamy Arumugam - Infosys Advisory Group • Chris Blask - AlienVault • John Allen – IEIA Forum • Andy Bochman - IBM • Joel Garmon - Wake Forest Baptist Medical Center • Jason Christopher - FERC (CISO) • Art Conklin - University of Houston • Dr. Emannuel - Hooper Global Info Intel and Harvard • Benjamin Damm - Silver Springs Network • Bill Hunteman • Anthony David Scott - Accenture • Jamey Sample - PG&E • Steve Dougherty - IBM Global Technology Services • Ido Dubrawsky - Itron Panel Member • Michael Echols - Salt River Project • Dr. Barbara Endicott-Popovsky - University of Washington Representation • Cliff Eyre - PNNL • Maria Hayden - Pentagon • Charles Reilly – Southern California Edison • Craig Rosen - PG&E Service • Scott Saunders - SMUD • Chris Sawall - Ameren Gov't • Paul Skare - PNNL • Clay Storey - Avista Industry • Dan Thanos - GE Digital Energy • Kevin Tydings - SAIC Research • Don Weber - InGuardians • Mike Wenstrom - Mike Wenstrom Development Partners Vendor • Nic Ziccardi - Network & Security Technologies

Security Testing & Smart Grid Source: Searle, Justin (2012) AMI Penetration Test Plan, National Electric Sector Cybersecurity Organization.

Current Draft OST Competency Model Constructs (Goal & Task Categories) Constructs (Goal & Task Categories) Manage the project Mitigate vulnerabilities • • Develop project plan Identify resources • • Monitor project plan Plan and document actions • Communicate actions Identify the critical vulnerabilities Understand and demonstrate impact • • Identify critical vulnerabilities Specify target-specific impact • • Analyze and map critical vulnerabilities Determine implications/plan response • • Develop and execute mitigation strategy Communicate impact Penetrate targets Educate team and clients • • Educate team Identify targets to penetrate • • Educate clients Analyze targets to penetrate • Develop and execute penetration strategy Exploit vulnerabilities • Infrastructure • Web and Applications • Other • Perform tasks in a safe and lawful fashion

Developing the Science of Competency Assessment: JOB PERFORMANCE MODELING

What is a competency? Broad Consistent Skills (consistency of performance) Ability (transfer across domains) Inconsistent Narrow Knowledge Shallow Deep (understanding of strategy or procedure) = Novice = Apprentice = Journeyman = Master Source: Tobey, D. H. et. al. (in press) Predictive Performance Modeling: An innovative approach to defining critical competencies that distinguish levels of performance," National Board of Information Security Examiners, Idaho Falls, ID, OST Working Group Report NBISE-OST-11-01

Defining proficiency at multiple levels for multiple roles Source: Tobey, D. H. et. al. (2011) Predictive Performance Modeling: An innovative approach to defining critical competencies that distinguish levels of performance," National Board of Information Security Examiners, Idaho Falls, ID, OST Working Group Report NBISE-OST-11-01

Critical Differentiation Analysis Task Criticality Low High Task Differentiation #1 … #1 … #2 … #2 … Low #3 … #3 … #4 … #4 … #5 … #5 … #1 … #1 … #2 … #2 … High #3 … #3 … #4 … #4 … #5 … #5 …

Defining the path to performance Masters Source: Tobey, D. H. et. al. (2011) Predictive Performance Modeling: An innovative approach to defining critical competencies that distinguish levels of performance," National Board of Information Security Examiners, Idaho Falls, ID, OST Working Group Report NBISE-OST-11-01 Apprentices Journeymen

Self-Assessment Instrument Knowledge (Understanding) • Learning Modes/Hours • Degree of understanding • Degree of difficulty Skill (Consistency) • Self-efficacy scale • Frequency scale • Performance scale Ability • Planning and Monitoring • Problem Solving

Individual Competency Profile: Radial Chart Views by Composite, Knowledge, Skill, and Ability Mitigate Vulnerabilities 10 8 6 4 Exploit Penetrate Targets vulnerabilities 2 0 Identify Vulnerabilities Performance Levels • Red: Low performance • Yellow : Borderline performance • Green : High performance

Individual Competency Profile: Drill-down (Penetrate targets) Foundational Tasks Composite Comparative score score Identify ownership of gateway devices (16.77) 83.8 Average Identify recon that is within project scope (15.63) 46.8 Low Search online sources for useful information about a target (15.45) 53.5 Average Differentiating Tasks (with weights) Analyze data found on compromised machines to enable exploitation Average 36.0 deeper into the network (24.02) Identify major assets subject to attacks (23.67) 87.2 High Identify targets for potential exploitation (23.67) 56.0 High Analyze data found on compromised machines for strategic value as 26.2 Low seen by a worst case attacker (23.60) Overall Score My Score 54.9 Average Knowledge Skill Ability Identify recon that is within project scope 86.5 27.4 47.3

Workforce Planning Source: International Center for Leadership in Education (ICLE) Rigor/Relevance Framework ( Daggett, 2000)