IDS Active Response Mechanisms: Countermeasure Subsytem for Prelude - PDF document

IDS Active Response Mechanisms: Countermeasure Subsytem for Prelude IDS Krzysztof Zaraska July 9, 2002

1 Introduction The number of security vulnerabilities in popular software forces system ad- ministrators to constantly keep track of new releases of patches from soft- ware vendors and exploit code from underground community or security re- searchers. In ideal case patch is available before exploit; this is however not a rule. Furthermore in real life situations patch not always can be applied on time due to various circumstances (e.g. known problems with the patch, administrator work overload, etc.). IDS signatures for a particular type of at- tack are usually available soon after the exploit is released; hence a properly maintained IDS can detect if someone tries to exploit a known vulnerability (also, certain IDS may be capable of detecting unknown exploits by means such as polymorphic shellcode detection). The concept of active response mechanisms or countermeasures in IDS is based on the idea of having an IDS capable of automatic reaction once a compromise threat is observed. The goal is to prevent further compromise by blocking the communication between attacker and attacked machine. It must be noted that if such a system is implemented or configured incorrectly it can be fooled into blocking legitimate traffic. 1.1 Techniques The following techniques can be used to reach our goals: RST emmision is a simple yet commonly used method. It is based on the fact, that a network error message (TCP RST for TCP protocol or ICMP Net/Host/Port unreachable packet for UDP protocol) when received by a TCP/IP stack causes the connection to be dropped. The concept is then to build and inject spoofed TCP RST or ICMP packet when packet sniffer detects an attack. Unfortunately, this technique has a number of limitations, most important of which are the race situ- ation between a packet generated by IDS and packets send by attacker and vulnerability to evasion. The issues related to RST emmision are discussed in detail in [3]. This technique is for example used by Snort with FlexResp extension. firewall update In this technique the IDS modifies the firewall ruleset adding a rule blocking traffic from the attacker. Example solutions: Guardian[4], SnortSam[7]. routing table update is similar to firewall update, the main difference is that a null (blackhole) route to attacking host is added on border 1

routers. signature–based firewall , e.g. Hogwash [6] is a hybrid solution integrat- ing firewall with an IDS. Basically it analyses passing traffic like IDS does and if a packet is considered dangerous, it is dropped or modified. 1.2 Limitations 1.2.1 Timing It is obvious that the application discussed here is time–critical, i.e. an ac- tive response mechanism reacting after a long time is not useful. We must note however, that IDS reacts after the dangerous event has taken place (this does not apply to signature–based firewalls). It means that even if latency is minimized, IDS may not be able to prevent actual compromise from hap- pening; e.g. when IDS detects a packet containing shellcode and sends RST packet to tear connection down, the shellcode will still be executed. There- fore blocking communication between target and attacker is rather aimed towards preventing attacker from using obtained shell on remote machine, or isolating a worm–infected machine to prevent it from infecting further hosts. A solution here would be to block attacker at the attack preparation stage, e.g. when a portscan is detected. However since port scans are easy to spoof this would lead to a self–inflicted DoS condition (see below). 1.2.2 Self–inflicted DoS IDS with active response subsystem using firewall update can be exploited by an attacker in order to cause a DoS condition. For example, if an IDS is configured to automatically block every IP address from which a port scan originates, an attacker can easily spoof a port scan from a hosts she wishes to be blocked. DNS servers are standard example here — spoofing port scan from them will cause them to be blocked at the firewall, and since many applications (both client and server side) actively depend on DNS, a serious service disruption can occur. A standard solution is to incorporate a white list , containing hosts which cannot be blocked, such as DNS servers, Kerberos servers, etc. It must how- ever be noted that attacker may start spoofing hosts non–critical to network infrastructure which are not whitelisted but users depend on — i.e. high– profile Web sites. The countermeasure is to expand the whitelist, to include also these sites. This however still allows the attacker to cause harm, e.g. by spoofing selected clients and causing them to be blocked. 2

Another danger is clogging firewall with inserted rules; since packet has to be matched against a larger number of rules the processing time increases, and thus firewall throughput decreases. A detailed analysis of firewall be- havior with increasing numbers of rules can be found in [8]. It should be however noted that according to this study the performance drops signifi- cantly when ruleset consists of several hundreds or more rules. Therefore by spoofing large number of hosts, attacker may actually bring firewall to stop. This problem can be prevented by using rules with limited lifetime (i.e. removed from ruleset after a given time) and/or limiting number of rules that can be inserted. 2 Countermeasures in Prelude IDS 2.1 About Prelude IDS Prelude is a hybrid IDS, designed to be modular, fast, and efficient. It is composed of several sensors, each having a different task (host based, network based), and distributed accross the network. A central Manager is responsible for collecting alerts from sensor, and outputting them in an user readable way (file logging, database logging). All modules are based on Prelude Library, libprelude which provides unified support for secure communication (using OpenSSL library), common data types (e.g. alert representation as defined by IDMEF standard) and handling of dynamically loadable plugins. Detailed description of Prelude can be found in [1]. 2.2 Design Principles The design of the countermeasure (CM) subsystem follows the main princi- ple of Prelude architecture, i.e. modularity. Therefore the subsystem was designed and implemented as consististing of many modules performing pre- defined set of functions and communicating over standarized interfaces. This will allow modules to be easily replaced or added, also by third party vendors, since creating new modules does not require knowledge of large parts of the source tree. Furthermore, elements of subsystem can be placed on different hosts in order to better reflect network topology, facilitate administration, and speed up reaction. This approach also facilitates future enhancing of the subsystem, by replacing existing modules or adding new ones. Following types of modules are introduced: triggering modules which are responsible for analyzing alerts reported by Prelude sensors and making decision if, and what action should be 3

taken for each alert. Subsystem can be equipped with many trigger- ing modules, using various decision making algorithms. For example, a specialized module may be used for triggering immediate reaction to high severity alerts (e.g. detecting of attempted buffer overflow ex- ploit against a known to be vulnerable machine) along with a module performing correlation analysis on alerts stacked in the database and reacting when a possibly dangerous trend is detected; in this example decisions are taken independenty by both modules. Also, introducing manual triggering mechanism requires only adding a new triggering module. communication modules which are responsible for communication be- tween various components of the subsystem. In particular they are re- sponsible for communication from triggering modules and action mod- ules (agents) performing actual action. action modules (agents) which are responsible for performing actual ac- tion, i.e. killing attacked process or reconfiguring firewall to block com- munication between attacker and victim hosts. It is also needed to incorporate protection against self–inflicted Denial– of–Service, so the subsystem cannot be used to cause harm to protected network (or, at least such abuse should be made as difficult as possible for the potential attacker). Next, subsystem should be flexible: configuration should be easily adapt- able to the network it is being deployed on to reflect its structure and local security policies. 2.3 Architecture The main principle of subsystem architecture can be defined as distributed detection, centralized decision and distributed reaction. This principle fol- lows Prelude’s semi–distributed architecture, where sensors located in the various parts of the guarded network send reports to a central manager (or, to a central network of managers if such a configuration is used). With such architecture it’s natural that decisions regarding using available countermea- sures should be taken at manager level; therefore the most natural solution is to implement triggering modules as Prelude Manager plugins. Also, sen- sors communicate with managers and managers communicate with agents; there’s no direct communication between sensors and agents. System behavior under attack is show in figure 1, data flow in this situa- tion is shown in figure 2. 4

Figure 1: Prelude with CM subsystem in action (based on [2]). Numbers in parantheses denote events sequence 5