PATCHING LESSONS LEARNED: An Interesting Perspective WHO WE ARE - - - PowerPoint PPT Presentation

Cyber Security | Compliance | Industrial Computing

PATCHING LESSONS LEARNED: An Interesting Perspective

WHO WE ARE -

MISSION

We ensure the success of our customers by developing innovative technology solutions that optimize and protect critical infrastructure

HISTORY

4 Company was started in 1981. Happy 36th birthday to us! 4 25+ years of experience in the design, manufacture, and lifecycle management of industrial HMIs and SCADA devices 4 11+ years of experience working with cyber security in the energy industry 4 ISO 9001:2008 certified 4 In 2013, we were awarded a $4.3M cooperative agreement by the Department of Energy in the Cybersecurity for Energy Delivery Systems (CEDS) Program 4 In 2014, we merged the computing and cyber security businesses under the name of FoxGuard Solutions 4 Total employees = ~110

Cyber Security | Compliance | Industrial Computing 2

www.foxguardsolutions.com

WHAT WE DO - FoxGuard serves as a solution provider for mission-critical applications in Critical Infrastructure and Key Resource (CIKR) sectors. Our Solutions Fall Into Three Primary Categories: 1. Cyber Security 2. Compliance 3. Industrial Computing Hardware

Cyber Security | Compliance | Industrial Computing 3

www.foxguardsolutions.com

PATCH & UPDATE MANAGEMENT PROGRAM -

Patch & Update Data Aggregator/Web Portal

4 Provides users a with single location to find information about patches and updates applicable to energy delivery industrial control system devices 4 The portal serves as a repository for Hash Authentication information, patch discovery evidence and device End of Support (EOS) documentation

Patch & Update Authentication

4 Our aggregated Hash Files from vendors provide users a central location to help verify the integrity of downloaded patches and updates prior to deployment 4 The Hash Authentication Tool allows customers receiving aggregated patch data via customized reports to authenticate that reports have not been compromised

Validation Techniques

4 Provides users with proven methodologies to validate patches and updates before deployment 4 Users may self-perform, set up their own validation lab, contract validation services or take a combined approach

Query Engine

4 The Query Engine will support multiple device types and across various energy delivery ICS vendors 4 Enables users to query IT and OT equipment to determine relevant baseline information such as Make, Model, Firmware Version and Serial Number – this information is critical for accurate patch discovery 4 The Query Engine will offer an easy to use user interface supporting a Patch Gap Analysis Dashboard simplifying the process of determining your current patch status and gaps Cyber Security | Compliance | Industrial Computing 4

www.foxguardsolutions.com

PATCH & UPDATE MANAGEMENT PROGRAM -

Cyber Security | Compliance | Industrial Computing 5

www.foxguardsolutions.com

FOXGUARD SECURITY SOLUTIONS AROUND THE WORLD

4 FoxGuard Solutions builds customized programs that function as an extension of the customer’s organization, enabling them to provide easy to use, easy to manage cyber security to their controls while ensuring their reliability. 4 FoxGuard is currently managing & 114 Companies 181 Sites 36 States 30 Countries

Cyber Security | Compliance | Industrial Computing 6

www.foxguardsolutions.com

COMPANIES USING FOXGUARD -

Cyber Security | Compliance | Industrial Computing 7

www.foxguardsolutions.com

• PATCHING. -

How Hard Can It Be? -

Harder than you think. Trust us. &

Cyber Security | Compliance | Industrial Computing 8

www.foxguardsolutions.com

WHAT IS A PATCH? -

P A T C H UPDATE UPGRADE FIRMWARE ENHANCEMENT SERVICE BULLETIN

– Feature Enhancements And / Or Security Patches – Focus Is On The Security Patches, As These Address Vulnerabilities To Their

Company (Not To Mention Compliance Requirements)

Cyber Security | Compliance | Industrial Computing 9 www.foxguardsolutions.com

WHAT NEEDS PATCHES & UPDATES - OPERATING 3RD PARTY SYSTEMS APPLICATIONS NETWORK DEVICES FIELD DEVICES SUPPORTED ASSETS

Cyber Security | Compliance | Industrial Computing 10 www.foxguardsolutions.com

WHY? - Why is Patch Management important? -

4 Energy Utilities are high-risk targets 4 Patches are crucial

4 189 known vulnerabilities in ICS in 2015* 4 26 had exploits available* 4 170 had patches available*

4 NERC CIP says so

4 CIP-007-6 R2.1, R2.2, R2.3, R2.4 4 Large fines for failure to comply

*Kaspersky Labs Industrial Control Systems Vulnerabilities Statistics

Cyber Security | Compliance | Industrial Computing 11

www.foxguardsolutions.com

THE STRUGGLE -

Current State

4 Existing solutions: 4 Are fragmented with limited coverage 4 Do not provide standardized actionable output 4 Have widely varying capability sets 4 Changing/increasing compliance scope

Cyber Security | Compliance | Industrial Computing 12 www.foxguardsolutions.com

COMPLIANCE VS. SECURITY -

Compliance -

Cyber Security | Compliance | Industrial Computing 13

www.foxguardsolutions.com

COMPLIANCE VS. SECURITY -

Compliance -

Cyber Security | Compliance | Industrial Computing 14

www.foxguardsolutions.com

COMPLIANCE VS. SECURITY -

Cyber Security | Compliance | Industrial Computing 15

www.foxguardsolutions.com

COMPLIANCE VS. SECURITY -

Compliance -

Cyber Security | Compliance | Industrial Computing 16

www.foxguardsolutions.com

COMPLIANCE VS. SECURITY - Security

4 Installing patches mitigates risks from vulnerabilities 4 Increased reliability of services may occur as a result of patching 4 “Air-gapped” is not enough 4 Compliance only mandates “security patches” be installed, but non-security

patches may offer functionality that leads to security features.

4 Near real-time application of patches and/or mitigation 4 Zero Day is a real thing. (Forever Day is a real thing too. Security holes that

remain unpatched. – Unfixed vulnerabilities – usually with legacy applications.)

Cyber Security | Compliance | Industrial Computing 17

www.foxguardsolutions.com

LESSON #1 - Understanding IT vs. OT is Critical

4 OT devices are not and SHOULD NOT be treated as IT devices

– Need expertise to understand and manage the differences &

4 Not all vendors report current patch status every month so you

must contact them directly. (Keep track of these vendors.)

4 Regular patch notification for some OT vendors is a new &

concept &

Cyber Security | Compliance | Industrial Computing 18 www.foxguardsolutions.com

LESSON #2 - Public AND Private Patches – Which are Which?

4 Approximately half of all EDS have “Private” patches 4 Not readily available on the Internet 4 Must securely track credentials 4 Understand requirements for obtaining this information 4 Private Portal, Current Support Contract, Call,

Newsletter, etc.

Cyber Security | Compliance | Industrial Computing 19 www.foxguardsolutions.com

LESSON #3 - Patch Analysis Accuracy is Difficult -

4 Documenting the process (where to go, how to mine, etc.) is an

• n-going effort

4 The mining procedure changes with each vendor 4 Vendors are known to change their process 4 Some products can be intricate and time consuming to find details 4 Ex. Cisco – 4 Security bugs are listed individually 4 Security ratings are buried within Release Notes 4 Multiple CVE’s may be listed

Cyber Security | Compliance | Industrial Computing 20

www.foxguardsolutions.com

LESSON #3 -

Cyber Security | Compliance | Industrial Computing 21

www.foxguardsolutions.com

LESSON #4 - Asset Analysis is Complicated

4 On-Going effort – Keeping it “current” is hard 4 Sub-components 4 Numerous distinct software packages on HMI/SCADA devices 4 Vendor Ownership of Asset (may have many vendors that distribute a &

product) &

4 Support Provider (may be different from initial Vendor) 4 Obtaining sufficient information in order to patch properly 4 IT Contractors may help but are not bullet proof 4 Aggregate lists may not be sufficient – Each asset may be different, even if

it seems the same (i.e., serial number, patch status)

Cyber Security | Compliance | Industrial Computing 22

www.foxguardsolutions.com

LESSON #4 -

Cyber Security | Compliance | Industrial Computing 23

www.foxguardsolutions.com

LESSON #5 - Security vs. Non-Security Determination takes Knowledge -

4 Not all vendors provide security rating 4 Need the resources and expertise to determine, if a vendor doesn’t

provide

4 CVE Information and CVSS Scores are helpful, but are not always

provided &

4 Vulnerabilities are not always updated by the vendor

Cyber Security | Compliance | Industrial Computing 24 www.foxguardsolutions.com

LESSON #6 - A Patch is not a Patch is not a Patch

4 Several types of patches: 4 Cumulative – One installation includes previous installation (Microsoft’s

new model)

4 Independent – Each patch can be installed independent from others

(Current patch status may be difficult to track)

4 Primary/Dependent – Must have a previous installation before installing

the latest (Keep track of what installed)

4 Patches can be changed/rereleased/retracted 4 Back Dating patches is possible (Vendor releases today but is dated three

months ago.)

Cyber Security | Compliance | Industrial Computing 25 www.foxguardsolutions.com

LESSON #6 -

October 24, 2016 Three Patches December 2, 2016

Cyber Security | Compliance | Industrial Computing 26 www.foxguardsolutions.com

LESSON #6 -

October 24, 2016 Four Patches December 28, 2016

Cyber Security | Compliance | Industrial Computing 27 www.foxguardsolutions.com

LESSON #7 - Maintaining Evidence is Hard

4 Types of Documentation 4 Patch Documentation 4 Proof you checked 4 Record status at the time you checked 4 EOS Documentation 4 Not every product lives forever 4 Audit-ready documentation must be created 4 Indicate change in status over the course of product lifetime 4 Time Consuming 4 What is appropriate – “Audit-worthy” 4 Maintaining – Keeping up to date, as well as storage of documents 4 Audit trail

Cyber Security | Compliance | Industrial Computing 28 www.foxguardsolutions.com

LESSON #8 - Your Calendar with Many Sources is Timely to Manage -

Cyber Security | Compliance | Industrial Computing 29 www.foxguardsolutions.com

LESSON #9 - Time & Resource Intensive

4 Patch Discovery takes MANY hours every month 4 Contact vendors 4 Who to call? 4 Email? 4 Website? 4 Newsletter? 4 Automatically provided? 4 Maintain timeline for each vendor – 35 days? 4 Documentation, documentation, documentation

Cyber Security | Compliance | Industrial Computing 30 www.foxguardsolutions.com

LESSON #10 - Validation is Intricate

4 Scope 4 What to test, What level of detail (registry change applied to which

patch, file versions, specific changes in a shared XML file, etc.

4 Resources 4 People -Need the right people with the right training to understand this

process

4 Equipment – Needs to have access to a representative system

(production/lab, physical/virtual

4 Test Equipment – Need equipment different from what you need in &

production in order to test the system &

Cyber Security | Compliance | Industrial Computing 31 www.foxguardsolutions.com

LESSON #10 - Validation Standard

Cyber Security | Compliance | Industrial Computing 32 www.foxguardsolutions.com

WHY YOU SHOULD CARE? - The Risks:

4 Safety: 4 Improperly Patched - “Brick” a device or Create a false sense of

security

4 Temporal Vulnerability with a missed patch – The longer you go

without patching, the greater your vulnerability/risk

4 Reliability: 4 Impact on The Grid? Reliability is critical. 4 Efficiency: 4 Stay focused on the job at hand and leave patching to the experts 4 Compliance Risks – Fines for non-compliance

Cyber Security | Compliance | Industrial Computing 33 www.foxguardsolutions.com

ONE MORE TIME - The Lessons:

4 Understanding IT vs. OT is Critical 4 Public AND Private Patches – Which are Which? 4 Patch Analysis Accuracy is Difficult 4 Asset Analysis is Complicated 4 Security vs. Non-Security Determination takes Knowledge 4 A Patch is not a Patch is not a Patch 4 Maintaining Evidence is Hard 4 Your Calendar with Many Sources is Timely to Manage 4 Time & Resource Intensive 4 Validation is Intricate

IS YOUR PROGRAM ORGANIZED? ARE YOU READY?

Cyber Security | Compliance | Industrial Computing 34 www.foxguardsolutions.com

THE BENEFITS OF A HEALTHY PATCHING PROGRAM

End User Benefits

4 Centralizes patch and update information 4 Supports programmatic equipment querying using automation and a common toolset 4 Simplifies association between software and available patches / updates 4 Lower cost with scalability of the program 4 More accurate information 4 Recurring delivery of information

Cyber Security Advancements

4 Promotes end user awareness around patching, presence of vulnerabilities and

change management processes &

4 Provides common security classification in absence of vendor classification 4 Considers named sub-components and libraries to provide more comprehensive

security assessment &

4 Reduces likelihood of incorrect patch application 4 Standardizes presentation of patch information to end user

Cyber Security | Compliance | Industrial Computing 35 www.foxguardsolutions.com

QUESTIONS? -

QUESTIONS?

FoxGuard Solutions Offers:

4 Patch Management 4 Security Services 4 Integrated Security Solutions 4 Industrial Computers 4 And Many Other Custom Solutions

For more information please contact:

Michele Wright Lindsey Hale Product Manager Program Manager (540) 382 – 4234 x244 (540) 382 – 4234 x108 mwright@foxguardsolutions.com lhale@foxguardsolutions.com Cyber Security | Compliance | Industrial Computing 36

www.foxguardsolutions.com

NEED MORE HELP -

Cyber Security | Compliance | Industrial Computing 37

www.foxguardsolutions.com

WANT TO LEARN MORE?