[PPT] - OSMOSIS Open Source Monitoring Security Issues HACKITO ERGO SUM PowerPoint Presentation

SLIDE 1

OSMOSIS

Open Source Monitoring Security Issues HACKITO ERGO SUM 2014 / April 2014 / Paris

SLIDE 2

AGENDA

§ Who are we? § Open Source Monitoring Software § Results § Demonstration § Responses § Mitigations and conclusion

4/25/14 2 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 3

DEUTSCHE TELEKOM PROFILE

4/25/14 3 – Public – Deutsche Telekom AG / OSMOSIS

COSTUMERS & MARKETS FACTS & FIGURES

Telekom in figures

§ Revenue € 58.7 bn § Adjusted Ebitda

€ 18.7 bn

§ Free cash flow

€ 6.4 bn

§ Among the top100

companies worldwide (#75 in 2012 Fortune500 list)

Employees & responsibility

§ Employees worldwide:

235, 000

§ 9 ,000 trainees und cooperative

degree students in Germany

§ Pioneer of social issues

(pomotion of woman, data privacy, climate protection etc.)

Customers

§ >141 m mobile customers § >32 m fixed-line customers/

>17 m broadband customers

§ rd. 3 m (IP) TV customers

§ About 2 m workstation systems

marketed

Markets

§ Presence in 50 countries

§ Deutschland, Europa, USA:

using our own infrastructur

§ T-Systems:

globale presence & alliances via partners

Source: DT annual report to shareholders 2012/TMUS annual report to shareholders 2012

SLIDE 4

DEUTSCHE TELEKOM GROUP INFORMATION SECURITY

4/25/14 4 – Public – Deutsche Telekom AG / OSMOSIS

Intelligente Netzlösungen §

Security requirements

§

Privacy & Security Assessment (PSA)

§

Deutsche Telekom Cyber Emergency Response Team (CERT)

§

Implementation of measures

§

Technology

§

Testing

§

Abuse-Handling

Security levels

Security strategies

Standards

Incident management

Consulting

Innovation

Security requirements

SLIDE 5

OPEN SOURCE MONITORING SOFTWARE OVERVIEW

SUMMARY

§ Critical function in a corporate network § Lets you know how well the network is running § End-to-end monitoring for services up to detailed hardware view

JOINT FUNCTIONS IN THIS CASE

§ Web based solution § Agent based

OUT OF SCOPE

§ No IDS / IPS § No commercial solutions § No security monitoring

4/25/14 5 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 6

OPEN SOURCE MONITORING SOFTWARE THREATS

§ Ubiquitous component in network environments § Centralized access to multiple networks § Usually position deep in the internal network (as in: semi-trusted network) § Used in nearly each environment (from small business, over mid range up to enterprises) § MTAACA (machine that acts as client attack) and CTAMTAACA (clients that access machines that act as clients attack)

4/25/14 6 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 7

OPEN SOURCE MONITORING SOFTWARE RISKS

§ A more valuable target than perimetric systems § Input data parsing (logfiles, SNMP, traps, ...) § Web GUIs (OWASP Top 10 anyone?) § Some have home-brew agents – on EVERY system § Potential access to a lot of components in the perimeter and internal network

4/25/14 7 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 8

OPEN SOURCE MONITORING SOFTWARE HOW IS IT IMPLEMENTED TYPICALLY?

4/25/14 8 – Public – Deutsche Telekom AG / OSMOSIS

SNMP O W N C H E C K S

SLIDE 9

OPEN SOURCE MONITORING SOFTWARE WHAT WE COVERED

§ This is not an academic talk - we are talking about actual experience § Open Source tools are easy to audit (kinda) § Everyone has the chance to audit their own solution § Focus on market leading / industry standard software

4/25/14 9 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 10

OPEN SOURCE MONITORING SOFTWARE WHAT WE DID NOT COVER

§ No commercial / closed source solutions § Architectural software flaws § Critical “features” which should be disabled anyways

e.g. nrpe.cfg dont_blame_nrpe

§ No additional plugins, features , add-ons § Not the (home brewed) agents itself

4/25/14 10 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 11

OPEN SOURCE MONITORING SOFTWARE TOOLS WE COVERED

§ CACTI

“… network graphing solution …”; “… frontend is completely PHP driven …” src: http://www.cacti.net

§ NAGIOS

“Nagios Is The Industry Standard In IT Infrastructure Monitoring” src: http://www.nagios.org/

§ CHECK_MK (NAGIOS ADD-ON)

“Check_MK is a comprehensive add-on for the famous Open Source monitoring software Nagios …” src: https://mathias-kettner.com/check_mk_introduction.html

§ ICINGA

“Icinga is an enterprise grade open source monitoring system …” src: https://www.icinga.org/

4/25/14 11 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 12

OPEN SOURCE MONITORING SOFTWARE PUBLICLY KNOWN INCIDENTS

4/25/14 12 – Public – Deutsche Telekom AG / OSMOSIS

CVE2012-096 – Remote Buffer Overflow Nagios Hetzner (06/2013)

SLIDE 13

OPEN SOURCE MONITORING SOFTWARE OTHER INTERESTING INFORMATION

4/25/14 13 – Public – Deutsche Telekom AG / OSMOSIS

Public Buffer Overflow in CACTI (since 10/2013) NRPE - Remote command exec (04/2014)

SLIDE 14

RESULTS OVERALL

§ Critical issues were found in ALL audited solutions …

§

Memory corruption – Buffer/Heap Overflows

§

Off-by-one’s

§

CSRF

§

XSS

§

eval-processing untrusted input

§

Remote Code Execution

§

Arbitrary file access § Many web based bugs, as all the solutions use web GUIs

4/25/14 14 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 15

(Cacti)

Version 3.5.0b 1.9.1b 1.2.2p2 0.8.8a Number of Findings 1 2 7 3 CVSS 2 Score (highest finding) 4.9

AV:N/AC:M/Au::S/C:P/I:N/A:P

8.5

AV:N/AC:M/Au:S/C:C/I:C/A:C

8.5

AV:N/AC:M/Au:S/C:C/I:C/A:C

8.5

AV:N/AC:M/Au:S/C:C/I:C/A:C

Criticality medium high high high

Number of open findings 1* 1** 3

Announcement to vendor / developer 5th Dec. 2013 2nd Dec. 2013 8th Oct. 2013 15th Oct. 2013 Bug Fix Release 3.5.x*, 4.0.3 1.10.2, 1.9.4, 1.8.5 or latest release

1.2.4p1, 1.2.5i2 or latest release

n/a Public DTAG CERT Advisory

DTC- A-20140324-004 DTC- A-20140324-003 DTC- A-20140324-002 DTC- A-20140324-001

Remarks

* Bug fixes in the source code only

available. No updates

release available. ** exec of python code within WATO

RESULTS DETAILED VIEW

03.04.2014 15 – Confidential – Christian Sielaff / OSMOSIS

SLIDE 16

DEMONSTRATION

CAN WE GET A SHELL?

4/25/14 16 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 17

DEMONSTRATION NETWORK OVERVIEW

03.04.2014 17 – Confidential – Christian Sielaff / OSMOSIS

Hacker Terminal Server Cacti / Check_MK Administrator

SLIDE 18

DEMONSTRATION CACTI

03.04.2014 18 – Confidential – Christian Sielaff / OSMOSIS

Hacker Cacti Administrator Bugs:

§

cross site request forgery

§

command like exec

SLIDE 19

DEMONSTRATION CACTI

03.04.2014 19 – Confidential – Christian Sielaff / OSMOSIS

Hacker Cacti Administrator Bugs:

§

cross site request forgery

§

command like exec Get executed on Cacti server if:

§

Administrator clicks on a link or

§

Visit a malicious web site

SLIDE 20

DEMONSTRATION CACTI

03.04.2014 20 – Confidential – Christian Sielaff / OSMOSIS

Hacker Cacti Administrator Bugs:

§

cross site request forgery

§

command like exec Get executed on Cacti server if:

§

Administrator clicks on a link, or

§

Visit a malicious web site Pro:

§

Get a shell Con:

§

Need to know the Cacti URL

§

Admins needs to access link or site with link to trigger exploit

§

Outgoing connections my be restricted

§

Admins needs to logged in

SLIDE 21

DEMONSTRATION CACTI

03.04.2014 21 – Confidential – Christian Sielaff / OSMOSIS

Hacker Cacti Administrator Bugs:

§

cross site request forgery

§

command like exec Get executed on Cacti server if:

§

Administrator clicks on a link, or

§

Visit a malicious web site Pro:

§

Get a shell Con:

§

Need to know the Cacti URL

§

Admins needs to access link or site with link to trigger exploit

§

Outgoing connections my be restricted

§

Admins needs to logged in … not really let’s brute force the Admin ac account J

SLIDE 22

DEMONSTRATION CHECK_MK

03.04.2014 22 – Confidential – Christian Sielaff / OSMOSIS

Hacker Terminal Server Check_MK Administrator

SLIDE 23

DEMONSTRATION CHECK_MK

03.04.2014 23 – Confidential – Christian Sielaff / OSMOSIS

Hacker Terminal Server Check_MK Administrator Bugs:

§

cross site request forgery

§

command like exec

§

cross site scripting

SLIDE 24

DEMONSTRATION CHECK_MK

03.04.2014 24 – Confidential – Christian Sielaff / OSMOSIS

Hacker Terminal Server Check_MK Administrator Bugs:

§

cross site request forgery

§

command like exec

§

cross site scripting What is the problem:

§

Exploits a feature in WATO

§

Uploads and exec a snapshot

§

Snapshot contains plain python code

SLIDE 25

DEMONSTRATION CHECK_MK

03.04.2014 25 – Confidential – Christian Sielaff / OSMOSIS

Hacker Terminal Server Check_MK Administrator Bugs:

§

cross site request forgery

§

command like exec

§

cross site scripting What is the problem:

§

Exploits a feature in WATO

§

Uploads and exec a snapshot

§

Snapshot contains plain python code Pro:

§

Get a shell Con:

§

Need to know the Check_MK URL

§

Admins needs to access link or site with link to trigger exploit

§

Outgoing connections my be restricted

§

Admins needs to logged in

SLIDE 26

DEMONSTRATION CHECK_MK

03.04.2014 26 – Confidential – Christian Sielaff / OSMOSIS

Hacker Check_MK Administrator Bugs:

§

cross site request forgery

§

command like exec

§

cross site scripting What can we do better?

§

Use the agent on a system

§

Re-use existing connections Terminal Server

SLIDE 27

DEMONSTRATION CHECK_MK

03.04.2014 27 – Confidential – Christian Sielaff / OSMOSIS

Hacker Check_MK Administrator Bugs:

§

cross site request forgery

§

command like exec

§

cross site scripting What can we do better?

§

Use the agent on a system

§

Re-use existing connections Terminal Server

SLIDE 28

DEMONSTRATION CHECK_MK

03.04.2014 28 – Confidential – Christian Sielaff / OSMOSIS

Hacker Check_MK Administrator Bugs:

§

cross site request forgery

§

command like exec

§

cross site scripting What can we do better?

§

Use the agent on a system

§

Re-use existing connections Terminal Server Pro:

§

Get a shell

§

URL is no longer needed

§

Administrator not need a link to click

§

Triggers when the Administrator logs in

§

Using existing connections Con:

§

Need (privileged) access to a monitored system

SLIDE 29

DEMONSTRATION CHECK_MK

03.04.2014 29 – Confidential – Christian Sielaff / OSMOSIS

Hacker Check_MK Administrator Bugs:

§

cross site request forgery

§

command like exec

§

cross site scripting What can we do also?

§

Just a simple SSH login? Terminal Server

SLIDE 30

DEMONSTRATION CHECK_MK

03.04.2014 30 – Confidential – Christian Sielaff / OSMOSIS

Hacker Check_MK Administrator Bugs:

§

cross site request forgery

§

command like exec

§

cross site scripting What can we do also?

§

Just a simple SSH login?

§

A XSS triggers a CSRF triggers an … Terminal Server

SLIDE 31

DEMONSTRATION CHECK_MK

03.04.2014 31 – Confidential – Christian Sielaff / OSMOSIS

Hacker Check_MK Administrator Bugs:

§

cross site request forgery

§

command like exec

§

cross site scripting What can we do also?

§

Just a simple SSH login?

§

A XSS triggers a CSRF triggers an upload that triggers a shell J Terminal Server

SLIDE 32

DEMONSTRATION CHECK_MK

03.04.2014 32 – Confidential – Christian Sielaff / OSMOSIS

Hacker Check_MK Administrator Bugs:

§

cross site request forgery

§

command like exec

§

cross site scripting What can we do also?

§

Just a simple SSH login?

§

A XSS triggers a CSRF triggers an upload that triggers a shell J Terminal Server Pro:

§

Get a shell

§

URL is no longer needed

§

Administrator not need a link to click

§

Triggers when the Administrator logs in Con:

§

Logwatch feature (default installation is fair)

§

Outgoing connections my be restricted

SLIDE 33

DEMONSTRATION

CAN WE GET A SHELL? … YES J

4/25/14 33 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 34

RESPONSES CONTACT AND TIMELINES

CONTACTING

§ some developer without a contact option (expect a public mailing list – is this a good idea in such case?) § usually an Email contact is possible – also with a privacy option § Only Icinga provides an option for a private information sharing

http://www.icinga.org/faq/how-to-report-a-bug/#securityissue

TIMELINE

§ approximately six days from first response to a bug fix release – well done! § up to 85 days to a bug fix release § up to nothing until now L

ADVISORIES

§ post flaws to Bugtraq on 24th of March § get first responses regarding open findings 28th / 31st of March

4/25/14 34 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 35

RESPONSES FEEDBACK

§ „WHAT IS OWASP?"

It’s 2014, guys!

§ „THIS IS A FEATURE"

Yes, and a backdoor!

§ „WHAT TOOLS DID YOU USE FOR SCANNING?“

Hint: None, we had the source code – Duh!

§ „WHY SHOULD WE FIX WHAT YOU SEE AS A SECURITY PROBLEM? WE NEVER ASKED FOR THIS AUDIT!“

Approximately Right. Remember it’s open source? Open as in: I audit this code as much as I want to?

§ „-„

As in: No response at all after issues were committed to developer.

4/25/14 35 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 36

RESPONSES DISCLOSURE

SECURITY FIXES

§ Change log or Release notes _never_ mentions security fixes explicitly § No hints or information on the developer Web sites! § CVE _Common_ – never heard about that

CREDITS

§ What’s that?

BUT THERE ARE SOME PROFESSIONALS

§ The Icinga Team has published bug fix releases (incl. back ports), ordered CVE numbers and assign the issues as

Security issues. MANY THANKS AND WELL DONE!

4/25/14 36 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 37

MITIGATIONS BEST PRACTICES

BEST PRACTICES

§ Consider Icinga and Nagios Security Guidelines

e.g. http://docs.icinga.org/latest/en/security.html

§ Nothing similar available for Cacti and Check_MK

GENERAL BASICS

§ Patching and regular updates § OS and middleware hardening § Minimal rights on application level, but also on operating system level § Remove critical features (e.g. WATO in Check_MK) § Passwords

4/25/14 37 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 38

MITIGATIONS SEGREGATION

ON NETWORK LEVEL

§ Do not place such systems flat in your corporate network § Consider segregation based on functions, e.g. own monitoring systems for dedicated services § No internet for the admin workstations and monitoring system (incl. ICMP, DNS, NTP, … )

ON APPLICATION LEVEL

§ Segregate user and roles

4/25/14 38 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 39

MITIGATIONS ARCHITECTURE

AGENT BASED MONITORING

§ Needs privileged rights to get all information and listen to the network (often unauthenticated) § Security of agents should be discussed separately e.g. http://www.securityfocus.com/archive/1/531063/30/0/threaded

CHECK VIA SSH

§ Must be secured carefully via SSHd configuration – otherwise direct shell login

SOLUTION

§ Change the communication direction § Based on Check_MK’s agent, it’s just a configuration – no additional software needed

4/25/14 39 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 40

MITIGATIONS ARCHITECTURE

HOW IT WORKS

§ Run Check_MK agent locally and pipe output to a file

4/25/14 40 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 41

MITIGATIONS ARCHITECTURE

HOW IT WORKS

§ Run Check_MK agent locally and pipe output to a file § Secure transfer, e.g. via SCP/SFTP

4/25/14 41 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 42

MITIGATIONS ARCHITECTURE

HOW IT WORKS

§ Run Check_MK agent locally and pipe output to a file § Secure transfer, e.g. via SCP/SFTP § Configure Check_MK Configuration & Check Engine to get information from a local file

4/25/14 42 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 43

MITIGATIONS ARCHITECTURE

4/25/14 43 – Public – Deutsche Telekom AG / OSMOSIS

O W N C H E C K S

SLIDE 44

CONCLUSION

§ Take care about your used solutions incl. additional features, add-ons, plug ins, self written checks and architecture. § When it named Open Source, it does not mean it is secure itself! § In general Open Source Monitoring solutions are not more or less secure than commercial ones. § Strong isolation of administrator workstations and your monitoring system as well. § @Developer: Check OWASP regularly!

4/25/14 44 – Public – Deutsche Telekom AG / OSMOSIS

SLIDE 45