Towards a Secure and Resilient Industrial Control System Using - - PowerPoint PPT Presentation

Towards a Secure and Resilient Industrial Control System Using Software-Defined Networking

Dong (Kevin) Jin

1

Who am I?

• CS faculty, Ph.D., University of Illinois at Urbana-

Champaign (UIUC), http://cs.iit.edu/~djin/

• Research: cyber-security, networking, cyber-physical

system security, simulation & modeling

• Industrial experience at Los Alamos National Lab, IBM,

Motorola

• I like designing/building/deploying large-scale

software systems that are grounded in strong theoretical principles

2

Master of Cyber Security Program

• New CS master degree started at Fall 2019
• What is unique?

– From theory to practice

• Data and information security
• Network and system security
• Software security
• Why join us?

– IIT is a Center of Academic Excellence in Information Assurance Education (CAE/IAE) designated by the National Security Agency – CS

• Multi-millions of federal/industrial research grants in cyber security
• A very strong team in the cyber security research and education
• How to join?

– https://science.iit.edu/programs/graduate/master-cybersecurity- mcybcode

– Also available to co-terminal students

3

Master of Cyber Security Program

Illinois Tech Team HackIllinois 2019@UIUC

4

Research Areas and Projects

Looking for strong and self-motivated students to work together!

More Details: http://cs.iit.edu/~djin/research/index.html

5

How to Get Involved in Our Research

• For all students, excellent performance in

– CS 458 Information Security – CS 558 Advanced Computer Security (Semester-long project) – CSP 544 System and Network Security (Hands-on Labs) – new in Spring 2020

• Master students

– CS 597 (Research Project), semester-long projects for credits – CS 591 (Master thesis), typically two-semester commitment

• Undergraduate students

– CS 497 (undergraduate research) with me, semester-long projects for credits More Details: http://cs.iit.edu/~djin/research/opening.html

6

Industrial Control Systems (ICS)

7

• Control many critical infrastructures
• Modern ICSes increasingly adopt Internet technology

to boost control efficiency

Next Generation of Power Grid

LOADS SITES DISTRIBUTION TRANSFORMER DISTRIBUTION SUBSTATION TRANSMISSION GENERATION

More Efficient or More Vulnerable?

8

Picture source: NIST Framework and Roadmap for Smart Grid Interoperability Standards Distribution Ops Transmission Ops

Operations Service Providers Bulk Generation Distribution Customer Markets

RTO/ISO Ops

DMS Asset Mgmt

Enterprise Bus

EMS RTO SCADA EMS WAMS MDMS Demand Response Retailer/ Wholesaler

Transmission

ISO/RTO Participation Aggregator Energy Market Clearing hosue Market Services Interface Plant Control System Generators Substation Device Field Device Distributed Generation

Utility Provider Third-Party Provider

CIS Billing Home/Building Manager Aggregator Electric Vehicle Distributed Generation Electric Storage Appliances Thermostat Customer EMS Customer Equipment Meter Others CIS Billing Retail Energy Provider

Premises Networks

Energy Services Interface Metering System Distribution SCADA

Enterprise Bus

Transmission SCADA

Enterprise Bus Wide Area Network Substation LANs Internet / e-business Field Area Networks

Data Collector Substation Controller Electric Storage

Internet / e-business

Communication Path Network

Cyber Threats in Power Grids

9

Picture source:

• 1. National Cybersecurity and Communications Integration Center (NCCIC). ICS-CERT Monitor Sep 2014 – Feb 2015
• 2. http://dailysignal.com/2016/01/13/ukraine-goes-dark-russia-attributed-hackers-take-down-power-grid/
• 245 incidents,

reported by ICS-CERT

• 32% in energy sector

Ukraine Power Grid Cyber Attack

• 230,000 residents in

western Ukraine

• 6 hours, 73 MW power

lost in Dec 2015

Protection of Industrial Control Systems

10

• Commercial off-the-shelf products

– e.g., firewalls, anti-virus software – fine-grained protection at single device only

• How to check system-wide requirements?

– Security (e.g., access control) – Performance (e.g., end-to-end delay)

• How to safely incorporate existing networking

technologies into control systems?

– Real time operations – Large-scale networks – Lack of real testbed (unlike Internet)

11

Our Work: Enable a Secure and Resilient ICS in Microgrid with SDN

Power Grid Component Layer Power Network Layer Communication Network Layer SDN Control Layer Application Layer Control Management Monitoring

ICS – industrial control system SDN – software-defined networking

Contribution I A novel SDN architecture in microgrid

12

KH LS Faciliti es Vandercook I WH PH AM HH Library CR T N T C T S TBC LSR Main ERB SSV Machine ry Tower Cunningha m Gunsaulu s Bailey Carman North South Fowler East Grad Lewis Farr KappaASA TRI PKS ASP DTD SPE PKP

Pershing Substation (12.47 kV)

MTCC Eng 1 SB SH Plant ComEd ComEd Vanderco

• k II

Power Plant Heat Plant

N S

Loop 1 Loop 2 Loop 3 Loop 4 Loop 5 Loop 6 Loop 7

Fiber Optic Infrastructure – SDN Ring Fiber Optic Infrastructure – SDN Ring

Battery Storage

Fisk Substation (12.47 kV) Solar PV Gas Generator Charging Station Wind Turbine Building Sensors Major Distribution Points

Control Center Existing Master Controller Microgrid App SDN Master Controller Network App Local Controller 1 PMU Local Controller 2 Smart Building Local Controller n Communication Networks

Transition to an SDN-Enabled Microgrid

• Facility

– DOE-funded IIT Microgrid – First Cluster of Microgrids in US – SDN deployment

• Big data available

– Processing – Storage – Analytics

Simulation Testbed -> Living Lab In-house research idea -> Real system deployment

13

Power Grid Component Layer Power Network Layer Communication Network Layer SDN Control Layer Application Layer SDN Application IDS Verification Self-healing Network

ICS – industrial control system SDN – software-defined networking

Our Work: Enable a Secure and Resilient ICS in Microgrid with SDN

Control Management Monitoring

Contribution II Innovative SDN-based security applications

14

Power Grid Component Layer Power Network Layer Communication Network Layer SDN Control Layer Application Layer SDN Application IDS Verification Self-healing Network

ICS – industrial control system SDN – software-defined networking

Our Work: Enable a Secure and Resilient ICS in Microgrid with SDN

Control Management Monitoring

Contribution III SDN-enabled microgrid testbed

• Parallel Simulation

(scalability)

• Virtual-Machine-based

Emulation (fidelity)

• SDN Background
• Applications

– Network Verification[1] – Self-healing PMU system [2]

• Testing and Evaluation Platform[3]

[1] Wenxuan Zhou, Dong Jin, Jason Croft, Matthew Caesar, and Brighten Godfrey. “Enforcing Customizable Consistency Properties in Software-Defined Networks.” USENIX NSDI [2] Dong Jin, Zhiyi Li, Christopher Hannon, Chen Chen, Jianhui Wang, Mohammad Shahidehpour and Cheol Won Lee. "Towards a Cyber Resilient and Secure Microgrid Using Software-Defined Networking." IEEE Transactions on Smart Grid [3] Christopher Hannon, Jiaqi Yan and Dong Jin. “DSSnet: A Smart Grid Modeling Platform Combining Electrical Power Distribution System Simulation and Software Defined Networking Emulation.” ACM SIGSIM-PADS (Best Paper Finalist)

15

Outline

Picture Source: Nick McKeown, Open Networking Summit 2012

Specialized Control Plane Specialized Hardware Specialized Features

Closed, proprietary Slow innovation Open interfaces Rapid innovation

App App App App App App App App

Control Plane

Open Interface

Merchant Switching Chips

Open Interface

SDN Background

16

Software Defined Networks

control plane: distributed algorithms data plane: packet processing

17

decouple control and data planes

Software Defined Networks

18

decouple control and data planes by providing open standard API

Software Defined Networks

19

(Logically) Centralized Controller

Controller Platform

20

Protocols è Applications

Controller Platform Controller Application

21

SDN Architecture

Data Plane Control Plane Application Plane App 1 App 2 App n

22

…

• Logically centralized control
• Global view
• Direct network control

Network Verifier Updates

• SDN Background
• Applications

– Network Verification[1] – Self-healing PMU system [2]

• Testing and Evaluation Platform[3]

[1] Wenxuan Zhou, Dong Jin, Jason Croft, Matthew Caesar, and Brighten Godfrey. “Enforcing Customizable Consistency Properties in Software-Defined Networks.” USENIX NSDI [2] Dong Jin, Zhiyi Li, Christopher Hannon, Chen Chen, Jianhui Wang, Mohammad Shahidehpour and Cheol Won Lee. "Towards a Cyber Resilient and Secure Microgrid Using Software-Defined Networking." IEEE Transactions on Smart Grid [3] Christopher Hannon, Jiaqi Yan and Dong Jin. “DSSnet: A Smart Grid Modeling Platform Combining Electrical Power Distribution System Simulation and Software Defined Networking Emulation.” ACM SIGSIM-PADS (Best Paper Finalist)

23

Outline

Network Verification - Motivation

89% of operators

never sure that config changes are bug-free

Survey of network operators: [Kim, Reich, Gupta, Shahbaz, Feamster, Clark, USENIX NSDI 2015]

82%

concerned that changes would cause problems with existing functionality

5

24

Network Verification

25

Prior Work

• Static network snapshot

analysis

– Klee [2008] – Anteater [2011]

• Dynamic verification

– FlowChecker [2011] – VeriFlow [2012] – HSA [2012] – Sphinx [2015]

26

Switch'A' Switch'B' Controller' Remove&rule&1& Install'rule'2' rule%1% rule%2%

Challenge: Timing Uncertainty

Old config: A => B (rule 1) New config: B => A (rule 2)

Packet'

Challenge: Timing Uncertainty

27

Switch'A' Switch'B' Controller' Install'rule'2' rule%1% rule%2% Remove&rule&1& (delayed)&

Loop-freedom Violation

Uncertainty-aware Modeling

• Naively, represent every possible network state O(2n)
• Uncertainty-aware graph: represent all possible combinations

28

SDN-based Verification System

29

A

B Reduce search space Real-time requirement Traverse graph model (A can reach B)

Enforcing dynamic correctness with heuristically maximized parallelism

30

A should reach B

2 1 3 4

SDN-based Verification System

OK, but…

31

Can the system “deadlock”?

• Proved classes of networks that never deadlock
• Experimentally rare in practice!
• Last resort: heavyweight “fallback” like consistent updates

[Reitblatt et al, SIGCOMM 2012]

Is it fast?

25000$ 20000$ 15000$ 10000$ 5000$ 0$ 7/22/2014$ 22:00:00$ 7/22/2014$ 23:00:00$ 7/23/2014$ 0:00:00$ 7/23/2014$ 1:00:00$

//$ //$ //$ //$ //$ //$

Time$ Number$of$Rules$ in$the$Network$

7/22/2014$ 22:00:02$ 7/22/2014$ 23:00:02$ 7/23/2014$ 0:00:02$ 7/23/2014$ 1:00:02$

Immediate Update GCC Consistent Updates

Comple?on$ Time$

}

CCG

• SDN Background
• Applications

– Network Verification[1] – Self-healing PMU system [2]

• Testing and Evaluation Platform[3]

[1] Wenxuan Zhou, Dong Jin, Jason Croft, Matthew Caesar, and Brighten Godfrey. “Enforcing Customizable Consistency Properties in Software-Defined Networks.” USENIX NSDI [2] Dong Jin, Zhiyi Li, Christopher Hannon, Chen Chen, Jianhui Wang, Mohammad Shahidehpour and Cheol Won Lee. "Towards a Cyber Resilient and Secure Microgrid Using Software-Defined Networking." IEEE Transactions on Smart Grid [3] Christopher Hannon, Jiaqi Yan and Dong Jin. “DSSnet: A Smart Grid Modeling Platform Combining Electrical Power Distribution System Simulation and Software Defined Networking Emulation.” ACM SIGSIM-PADS (Best Paper Finalist)

32

Outline

Self-healing PMU Network

33

Source: https://www.naspi.org/sites

1700

3000+

200+ 200

North America

Production-grade

2007 2009 2014

By 2019

• Wide Area Measurement

Systems (WAMS)

• Synchrophasors, GPS time

stamping

PMUphasor measurement unit

Challenges

34

• High volume of measurement data
• Network architecture – no standard yet
• Cyber-attacks and human errors

– e.g., denial-of-service, man-in-the-middle attacks [1][2]

[1] C. Beasley, G. K. Venayagamoorthy, and R. Brooks. Cyber security evaluation of synchrophasors in a power system. [2] T. Morris, S. Pan, J. Lewis, J. Moorhead, N. Younan, R. King, M. Freund, and V. Madani. Cybersecurity risk testing of substation phasor measurement units and phasor data concentrators.

Control Center switch switch switch PDC PDC PMUs PMUs PMUs PMUs PMUs PMUs switch switch

WAN

PMUs PMUs PMUs Control Center PMU PMU PMU PDC PDC

Under attack Disconnected

PMU phasor measurement unit PDC phasor data concentrator

• Lose system observability
• Affect state estimation

Self-healing PMU network

35

• Objectives

– Recover power system observability

– Isolate compromised devices; re-connect uncompromised devices

– Fast recovery speed – Easy and inexpensive deployment

• Contributions

– An SDN-based architecture – Global-optimized self-healing solution – A working prototype system with good system performance

36

Power Network Layer PMU

PMU PMU PMU PDC PDC SDN Controller Control Center

PDC Control Layer Communication Network Layer

Self-healing PMU Infrastructure PMU network layer creation

Self-healing PMU network

System models Graph Gp( B, Lp) power transmission network Gc(U U D U R, Lc) IP-based PMU network

B - set of buses; Lp -set of transmission lines ; U - set of PMUs D - set of PDC; R - set of router; Lc -set of links

Power Network Layer PMU

PMU PMU PMU PDC PDC SDN Controller Control Center

PDC Control Layer Communication Network Layer

37

PMU/PDC application layer creation

Self-healing PMU network

Observability function of bus i where defines the bus connectivity

Power Network Layer PMU

3 2 1 PDC PDC SDN Controller Control Center

PDC Control Layer Communication Network Layer

38

Self-healing PMU network

Real Data Collected from IIT Distribution System PMU network

Control Center Monitoring System

PMU3 PMU1

39

Self-healing PMU network

Power Network Layer PMU

3 2 1 A B SDN Controller Control Center

PDC Control Layer Communication Network Layer

PDC A stop functioning under a cyber-attack

PMU3 PMU1

? ?

40

Self-healing PMU network

Power Network Layer PMU

3 2 1 A B SDN Controller Control Center

PDC Control Layer Communication Network Layer

? ?

Objective: quickly restore system power

• bservability

Stage I minimize # of reconnected PMUs Stage II minimize # of new rules on SDN switches Constraints

• PDC connection space constraints
• Congestion freedom constraints
• Rule capacity constraints

41

Self-healing PMU network

Power Network Layer PMU

3 2 1 A B SDN Controller Control Center

PDC Control Layer Communication Network Layer

?

Objective: quickly restore system power

• bservability

Stage I minimize # of reconnected PMUs Stage II minimize # of new rules on SDN switches Constraints

• PDC connection space constraints
• Congestion freedom constraints
• Rule capacity constraints

42

Self-healing PMU network

Power Network Layer PMU

3 2 1 A B SDN Controller Control Center

PDC Control Layer Communication Network Layer

?

Objective: quickly restore system power

• bservability

Stage I minimize # of reconnected PMUs Stage II minimize # of new rules on SDN switches Constraints

• PDC connection space constraints
• Congestion freedom constraints
• Rule capacity constraints

PMU3 - reconnected PMU1

• SDN Background
• Applications

– Network Verification[1] – Self-healing PMU system [2]

• Testing and Evaluation Platform[3]

[1] Wenxuan Zhou, Dong Jin, Jason Croft, Matthew Caesar, and Brighten Godfrey. “Enforcing Customizable Consistency Properties in Software-Defined Networks.” USENIX NSDI [2] Dong Jin, Zhiyi Li, Christopher Hannon, Chen Chen, Jianhui Wang, Mohammad Shahidehpour and Cheol Won Lee. "Towards a Cyber Resilient and Secure Microgrid Using Software-Defined Networking." IEEE Transactions on Smart Grid [3] Christopher Hannon, Jiaqi Yan and Dong Jin. “DSSnet: A Smart Grid Modeling Platform Combining Electrical Power Distribution System Simulation and Software Defined Networking Emulation.” ACM SIGSIM-PADS (Best Paper Finalist)

43

Outline

44

Test Systems in Lab

Security Exercise/Evaluation

• Scalable
• Flexible
• Controllable
• Reproducible
• No interference with real systems
• Realistic settings

A Large-scale, High-fidelity Simulation/Emulation Testbed

Testbed for Smart Grid Security

Testbed Design

45

• SDN Emulation

– lightweight virtual machine – unmodified code execution – virtual time system

• Parallel Simulation Engine

– 1 million nodes

• Simulation

– S3FNet: communication network – OpenDSS: power distribution system

• Using by

– IBM Research – Boeing – Argonne National Lab

Parallel Simulation VM Controller SDN Emulation VM1 VM0 VMn Cn C3 Global Scheduler VM2 VM3 Parallel Simulation Kernel

…

Network Simulator Power Simulator

Parallel Simulation/Emulation Testbed

[Best paper award, PADS’12], [Best paper finalist, PADS’16]

Cyber-security Evaluation

46

Extensively utilize the testbed to evaluate cyber-attacks

• Power grid control network

– supervisory control and data acquisition (SCADA)

• Wide area monitoring

– Phasor measurement unit (PMU)

• Advanced metering infrastructure (AMI)

– Demand response – Load disaggregation

• Transactive control networks

Use Case: DDoS Attack in Smart Meter Networks

• Amplification

– Increased volume of traffic

• Reflection

– Spoofed source address (the victim’s address)

47

C12.22 Trace Service

Meter Data Collector Victim Attackers spoof victim source address

S R1 R2 R3 26 26+R1 26+R1+R2 Request Response 26+R1+R2

Attacking Experiment

• 4x4 blocks, 448 meters
• ZigBee wireless network, 1 Mb/s

bandwidth

48

Attacking Experiment

49

• 4x4 blocks, 448 meters
• ZigBee wireless network, 1 Mb/s

bandwidth

• 5 attackers
• Victim: the single egress point

(meter gateway)

Experimental Results – Packet Loss

50

100 200 300 400 500 600 700 800 0 100 200 300 400 500 600 700 800 Y Coordinate (meter) X Coordinate (meter) 100 200 300 400 500 600 700 800 0 100 200 300 400 500 600 700 800 Y Coordinate (meter) X Coordinate (meter)

Normal Under DDoS Attack 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Packet Loss Rate

Conclusion

51

• Goal: To build a more secure, resilient, and safe

cyber-environment for industrial control systems

• Enable a cyber secure and resilient ICS in

microgrid with SDN

– A novel SDN architecture in microgrid – Innovative SDN-based security applications – Microgrid testbed using parallel simulation and virtual-machine-based emulation

52