Increase Datacenter Security Posture LinuxCon North America 2016 - - PowerPoint PPT Presentation

Using Hypervisor and Container Technology to Increase Datacenter Security Posture

LinuxCon North America 2016 – Toronto Canada

#whoami – Tim Mackey

Current roles: Senior Technical Evangelist; Occasional coder

• Former XenServer Community Manager in Citrix Open Source Business Office

Cool things I’ve done

• Designed laser communication systems
• Early designer of retail self-checkout machines
• Embedded special relativity algorithms into industrial control system

Find me

• Twitter: @TimInTech ( https://twitter.com/TimInTech )
• SlideShare: slideshare.net/TimMackey
• LinkedIn: www.linkedin.com/in/mackeytim

Understanding the Attacker Model

Attacks are Big Business

In 2015, 89% of data breaches had a financial or espionage motive

Source: Verizon 2016 Data Breach Report

Attackers Decide What’s Valuable …

But security investment is often not aligned with actual risks

Anatomy of a New Attack

Potential Attack Iterate Test against platforms Document

Don’t forget PR department!

Deploy

Exploiting a Vulnerability

Knowledge is Key. Can You Keep Up?

glibc

Bug Reported

July 2015 Vuln: CVE-2015-7547: glibc getaddrinfo stack-based buffer overflow

Knowledge is Key. Can You Keep Up?

glibc

Vuln Introduced

May 2008

glibc

Bug Reported

July 2015

CVE-2015- 7547 CVE Assigned

Feb 16-2016

Low Security Risk

Vuln: CVE-2015-7547: glibc getaddrinfo stack-based buffer overflow

Knowledge is Key. Can You Keep Up?

glibc

Vuln Introduced

May 2008

CVE-2015- 7547 CVE Assigned

Feb 16-2016

glibc

Bug Reported

July 2015

National Vulnerability Database

Vuln Published

Feb 18-2016

Moderate Security Risk Low Security Risk

Vuln: CVE-2015-7547: glibc getaddrinfo stack-based buffer overflow

Knowledge is Key. Can You Keep Up?

glibc

Vuln Introduced

National Vulnerability Database

Vuln Published You Find It

May 2008

CVE-2015- 7547 CVE Assigned

Feb 16-2016 Feb 18-2016

glibc

Bug Reported

July 2015

Patches Available You Fix It Highest Security Risk Moderate Security Risk Low Security Risk

Vuln: CVE-2015-7547: glibc getaddrinfo stack-based buffer overflow

Understanding Vulnerability Impact

500 1000 1500 2000 2500 3000 3500 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015

Open Source Vulnerabilities Reported Per Year

BDS-exclusive nvd

Reference: Black Duck Software KnowledgeBase, NVD

Vulnerability Disclosures Trending Upward

Virtualization Extensions for Threat Mitigation

Primary goals

• Protect against BIOS and firmware attacks
• Protect cryptographic host state
• Ensure valid hypervisor kernel
• Validate launch of critical VMs
• Attest to hosts’ trust state

Implemented by

• Intel Haswell and newer
• Cryptographic hashes stored in TPM

Intel TXT – Trusted Execution Protection - Foundational

Intel SMAP – Supervisor Mode Access Protection

Operating System Kernel User Mode Applications

Read Application Memory Write Application Memory Read Kernel Memory Write Kernel Memory Read Kernel Memory Write Kernel Memory Read Application Memory Write Application Memory

mov r8d,2Bh mov ss,r8w mov r9d,dword ptr [r13+3Ch] mov dword ptr [rsp],r9d mov esp,dword ptr [r13+48h] jmp fword ptr [r14] mov r14,rsp mov word ptr [rsp+8],23h mov word ptr [rsp+20h],2Bh mov r8d,dword ptr [r13+44h] and dword ptr [r13+44h],0FFFFFEFFh mov dword ptr [rsp+10h],r8d mov r8d,dword ptr [r13+48h] mov qword ptr [rsp+18h],r8 mov r8d,dword ptr [r13+3Ch] mov qword ptr [rsp],r8

Intel PML- Page Modification Logging

Intel PML- Page Modification Logging

Who changed the world? What in the world changed? When did the change occur? Why did the world change?

Intel EPT – Extended Page Tables

Page 0 … Page 13553 Page 13554 … … Page 126 Page 127 … Page 64589 Page 64590 Page 64591 Page 0 … Page 217 … Page 31289 … … Page 78924 … Page 97586 … 0→64589 13553→127 13554→64591 App Memory OS Memory TLB CR3

Virtual Machine

126→31289 127→0 64589→97586 64590→217 64591→78924 Host Memory EPT

Hypervisor

Hypervisor Memory Introspection – Enabled by EPT

Implementation Overview

• Critical memory pages are

assigned permissions in EPT

• Exception handler defined in

hypervisor

• Shadow EPT defined with elevated

privs

Protects Against Attack Techniques

• Rootkit injection
• Buffer overflow
• API hooking

VM Kernel Memory Layout

… Kernel Code (R/X) Driver Code (R/X) … Driver Data (R/W) Kernel Code (R/X) Kernel Data (R/W) … 126→31289 (R/X) 127→0 (R/X) 64589→97586 (R/W) 64590→217 (R/X) 64591→78924 (R/W)

EPT#1

126→31289 (+W) 127→0 (+W) 64589→97586 (+X) 64590→217 (+W) 64591→78924 (+X)

EPT#2 (Shadow) Exception Handler

Guest Guest Guest Guest Guest

Critical Memory Access Critical Memory Access Critical Memory Access Critical Memory Access Critical Memory Access

Networking Storage Compute

Simplified Hypervisor Introspection Architecture Diagram

Xen Project Hypervisor Control Domain (dom0) Security Appliance (domU) Memory Introspection Engine

Direct Inspect APIs

Virtual Switches as Local Edge Protection – Silent Block

Guest VM SSL access Attack silently blocked Virtual Switch Rules Ingress: HTTPS public Egress: Dynamic port to origin MySQL internal Private CIDR internal Port 22 access

Virtual Switches as Local Edge Protection – Traffic Monitor

Guest VM SSL access Attack blocked with traffic log Virtual Switch Rules Ingress: HTTPS public Egress: Dynamic port to origin MySQL internal Private CIDR internal Port 22 access

• vs Controller

Log SSH Port 22 access Create port mirror for attacker Traffic Monitor Virtual Switch Rules Ingress: HTTPS public Egress: Dynamic port to origin MySQL internal Private CIDR internal Mirror: Port 22 to Traffic Monitor All attacker traffic to monitor

Guest VM

Virtual Switches as Local Edge Protection – Quarantine

Guest VM SSL access Attack quarantined with full log Virtual Switch Rules Ingress: HTTPS public Egress: Dynamic port to origin MySQL internal Private CIDR internal Port 22 access

• vs Controller

Log SSH Port 22 access Create port mirror for attacker Quarantine VM for attacker use Trigger replacement VM for farm Traffic Monitor Virtual Switch Rules Ingress: HTTPS attacker Egress: Dynamic port to origin Mirror: Port 22 to Traffic Monitor All attacker traffic to monitor

Containers to Limit Scope of Compromise

Are Containers Production Ready?

Container Deployment Models

Container Use Cases

Application containers

• Hold a single application
• Can follow micro-services, cloud native design pattern
• Starting point for most container usage
• Short lifespan, many per host

System containers

• Proxy for a VM
• Insulate against core operating system
• Perfect for legacy apps
• Long lifespan, few per host

MySQL Tomcat nginx Kernel MySQL Tomcat nginx Kernel

Securing the Container Contents and Environment

Trust Container Source

Atomic Host Atomic App Atomic App Atomic Nulecule Atomic Nulecule RedHat Registry MySQL Redis Jenkins Docker Hub Docker Container Docker Container Docker Container Docker Container Docker Container Third Party and Custom Problem: Who to trust, and why?

• Trusted source?
• Unexpected image contents
• Locked application layer

versions (e.g. no yum update)

• Layer dependencies

(monolithic vs micro-services)

• Validated when?

Determine Who Can Launch A Container

Container default is root access

• RBAC/ABAC is orchestration specific

Docker Datacenter

• Universal Control Plane
• RBAC – LDAP/AD/local users
• Full/Restricted/View/None

Kubernetes

• Authorization modules
• Admission controllers

Define Sensible Container Network Policies

Docker default network is Linux Bridge Access policy defined in iptables

• Based on Docker daemon startup

External communication on by default

• - iptables=off to disable iptables modification

Inter container communication on by default

• - icc=false to disable inter container communication
• - link=CONTAINER_NAME_or_ID:ALIAS with EXPOSE ports from Docker file
• All inter-container/cross host communication is external

`docker network` command simplifies aspects of network design

• Create user defined networks, including overlay networks
• docker network create --driver bridge sql

Docker Networking - Example

Host eth0/10.204.136.1 Container veth0 Container veth1 Container veth2 Container veth3 Container veth4 Container veth5 docker0 NAT/ 172.16.1.0/24 iptables Host docker0 eth0/10.204.136.2 Container veth0 Container veth1 Container veth2 Container veth3 Container veth4 Container veth5 NAT/ 172.16.1.0/24 iptables

Host

Kubernetes Networking - Example

Kubernetes Network eth0/10.204.136.20 Pod Container Pause Container Container

veth0/10.204.136.21

Pod Container Pause Container Container

veth0/10.204.136.22

Host Kubernetes Network eth0/10.204.136.10 Pod Container Pause Container Container

veth0/10.204.136.11

Pod Container Pause Container Container

veth0/10.204.136.12

Limit the Scope of Compromise

• Enable Linux Security Modules
• SELinux
• -selinux-enabled on Docker engine, --security-opt=“label:profile”
• AppArmor
• - security-opt=“apparmor:profile”
• Apply Linux kernel security profiles
• grsecurity, PaX and seccomp protections for ALSR and RBAC
• Adjust privileged kernel capabilities
• Reduce capabilities with --cap-drop
• Beware –cap-add and –privileged=false, and CAP_SYS_ADMIN
• Use a minimal Linux Host OS
• Atomic host, CoreOS, RancherOS
• Reduce impact of noisy neighbors
• Use cgroups to set CPU shares and memory

Control Domain Networking Compute Storage Hypervisor Container VM

Minimal OS

Understanding Scope of Compromise – Protect From the Inside

Container Container Container

Container VM

Minimal OS

Container Container Container

Security Service

Container

Risk Mitigation Shrinks Scope of Compromise

Open source license compliance

• Ensure project dependencies are understood

Use of vulnerable open source components

• Is component a fork or dependency?
• How is component linked?

Operational risk

• Can you differentiate between “stable” and “dead”?
• Is there a significant change set in your future?
• API versioning
• Security response process for project

7 of the top 10

Software Companies (44 of the top 100)

6 of the top 8

Mobile Handset Vendors

6 of the top 10

Investment Banks

24

Countries

250+

Employees

1,800

Customers

Who is Black Duck Software?

27

Founded

2002

8,500

WEBSITES

350

BILLION LINES OF CODE

2,400

LICENSE TYPES

1.5

MILLION PROJECTS

76,000

VULNERABILITIES

• Largest database of open source project

information in the world.

• Vulnerabilities coverage extended through

partnership with Risk Based Security.

• The KnowledgeBase is essential for identifying

and solving open source issues.

Comprehensive KnowledgeBase

Black Duck Hub Security Architecture

Hub Scan

1

File and Directory Signatures

2

Open Source Component Identified

3

Hub Web Application Black Duck KnowledgeBase

On Premises Black Duck Data Center

We Need Your Help

Knowledge is power

• Know what’s running and why
• Define proactive vulnerability response process
• Don’t let technology hype cycle dictate security

Invest in defense in depth models

• Don’t rely on perimeter security to do heavy lifting
• Do look at hypervisor & container trends in security
• Make developers and ops teams part of the solution
• Focus attention on vulnerability remediation

Together we can build a more secure data center