Anomaly Based Intrusion Detection in Distributed Applications - - PowerPoint PPT Presentation

Anomaly Based Intrusion Detection in Distributed Applications without global clock

Eric Totel, Mouna Hkimi, Michel Hurfin, Mourad Leslous, Yvan Labiche SEC2-2016 5 July 2016

Outline of the Presentation

• Position of the problem
• Building a distributed application behavior model
• Partial Event Ordering
• Automaton recognizing sequences
• Temporal properties
• Applying Detection on an example
• Results on a distributed file system application

CentraleSupelec 2

Intrusion Detection in Distributed Systems

• Several nodes running processes
• Intrusion Detection Systems are deployed
• On the network (NIDS)
• On each node (HIDS)
• Local Detection of compromission
• No relationship between the states of the several nodes
• Alerts emitted takes into account the state of one node
• Current solutions
• Alert correlation: Requires total ordering of all alerts
• DIDS: Requires total ordering of all events analyzed
• In Cloud environments: virtual machines are often desynchronized (clock

drift)

CentraleSupelec 3

The Case of a Distributed Application

• How to enhance the detection ?
• Statement
• The states of the different nodes are not independent
• As such the behaviors of the different nodes of the application are not

independent

• The actions performed by the nodes are causally dependent on each other
• Local actions
• Messages exchanged
• Solution
• Build a reference model that takes into account the causal dependencies

between the nodes

• Without relying on a total ordering of the events (no global clock)

CentraleSupelec 4

Logs and partial ordering

• On each node, a process

produces a total ordered log

• Partial Ordering of the events on different nodes (Lamport happened

before relationship)

• e occured before f in the same log
• e is a message send and f its receipt
• there exists g such that
• How to learn the right sequences of actions performed by the

distributed processes ?

CentraleSupelec 5

8e 2 Eα, 8f 2 Eα, e α f Eα

i

e α g and g α f

Example of Logs

• A trace: two logs of two processes

and

• On this execution,
• No order relation between b and d:
• {a, b, d, c1!m, c1?m, e} is a valid sequence … but not the only one !

CentraleSupelec 6

Execution α Eα

1

Eα

2

1 a d 2 b c1?m 3 c1!m e

p1 p2 a b c1!m d c1?m e

a α b

Notion of a valid sequence

• Observed correct normal sequence
• Compliant with the partial relationship
• A sequence
• f events is valid iff

CentraleSupelec 7

Generation of valid sequences (1)

a b c1!m d c1?m e p1 p2

CentraleSupelec 8

Execution α Eα

1

Eα

2

1 a d 2 b c1?m 3 c1!m e

Generating the lattice of consistent cuts A valid sequence is a sequence

• f events consumed by a path

in the lattice of consistent cut

Generation of valid sequences (2)

a b c1!m d c1?m e p1 p2

CentraleSupelec 9

1 a 16 d 6 2 7 d 3 c1!m 13 c1!m 9 c1?m b 17 d e d b a

Generation of an automaton containing all the paths in the lattice of consistent cuts

Automaton from several executions

• CentraleSupelec

10

Execution α Execution β Eα

1

Eα

2

Eβ

1

Eβ

2

1 a d a f 2 b c1?m c1!m c1?m 3 c1!m e g

Merge the start states

• f all the automata

Analysis of the automaton

• Contains only the observed valid sequences
• In practice:
• In a heavy distributed application, it is very difficult to exhibit all the behaviors
• f the application due to concurrency
• It is thus very difficult to learn a complete behavior model
• Solution:
• Generalization of the automaton
• Permits to introduce new unlearned behaviors
• Ensures that all the original valid sequences are included in the generalized

automaton

CentraleSupelec 11

Generalization (k-tail algorithm)

• CentraleSupelec

12

• Advantage: can introduce new valid unlearned

sequences of events Disadvantage: can introduce incorrect sequences

• f events at the same time

k=1 (a low k permits a higher generalization)

How to deal with incorrect sequences ?

• Duality of models
• Automaton: exhaustive list of sequences
• Temporal properties: properties on the types of events
• Temporal invariants
• Issued from the domain of test
• Three invariants considered (a and b are event types)
• a is always followed by b
• a is never followed by b
• a always precedes b

CentraleSupelec 13

Invariants on our example

CentraleSupelec 14

• Model checking

(total of 59 invariants) Generalization

Duality of models

CentraleSupelec 15

Model Invariants that can be violated by the generalized automaton Generalized Automaton

• Non acceptable sequence {a, b, c1!m, d, c1?m, g}

(total of 10 invariants)

Valid/Accepted/Acceptable sequences

• Invariants computed on the
• riginal lattice of consistent cuts
• Invariants on valid sequences of

events

• Invariants are less restrictive than

the automaton

• We consider a sequence is

acceptable if it is accepted by the automaton and complies with the invariants

CentraleSupelec 16

∑' sequences accepted by the generalized automaton ∑'' acceptable sequences ∑ valid sequences

Detection algorithm

• Given a trace
• Is this trace compliant:
• With the generalized automaton
• With the temporal invariants
• Two strategies
• All total ordering of the events of the trace are compliant with the model
• At least one order of the events of the trace is compliant with the model
• In practice
• Strategy « all » is more time consuming
• Similar false positive rate in both approaches

CentraleSupelec 17

Simple Example: e-commerce

• 3 processes: article buying, 70 possible different behaviours

CentraleSupelec 18

Process (p2)

P2-P1!SEARCH P1-P2?AVAILABLE P2-P1!BUY P1-P2?SOLD P3-P1!SEARCH P1-P3?AVAILABLE P3-P1!BUY P1-P3?SOLD P2-P1?SEARCH P3-P1?SEARCH P1-P3!AVAILABLE P2-P1?BUY P3-P1?BUY P1-P2!SOLD P1-P3!SOLD

Server (p1) Process (p3)

P1-P2?AVAILABLE P1-P2!AVAILABLE

Detection Accuracy

• Simulations of an intrusion
• Removing an event
• Modifying the order of events
• Adding new events
• Violating the integrity of the distributed logs
• Are detected by the approach

CentraleSupelec 19

Generalization and False Positive Rate

CentraleSupelec 20

• Learning Phase with 10,

20, 30, 40, 50, 60 traces

• With a generalization

parameter k=1, 2, 3, 4, 5

• Result:
• The generalization

decreases the rate of the false positives, even with a low number of traces learnt

65% 70% 75% 84% 85% 19% 31% 39% 68% 71% 9% 16% 22% 42% 52% 6% 8% 10% 18% 30% 1% 3% 3% 9% 19% 0% 0% 0% 2% 8% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 1 2 3 4 5

k

False Positive Rate

traces learned=10 traces learned=20 traces learned=30 traces learned=40 traces learned=50 traces learned=60

Real World Evaluation: XtreemFS

• High Availability

Distributed Replicated File System

• Intrusion Detection

approach applied on a simple configuration of the nodes

CentraleSupelec 21

Experimentation applied

• Writing of a set of files
• 500 files used to learn the model
• 1640 files written to measure the false positive rate
• Traces obtained on each node by instrumenting the code of the file

servers

• One trace for a complete file write

CentraleSupelec 22

Model Size

• Number of traces used

to learn the model grow

• The number of

invariants lower

• The size (number of

states) of the automaton grows (k- tail applied with k=1)

CentraleSupelec 23

100 200 300 400 500 600 700 800 6200 6400 6600 6800 7000 7200 7400 7600 7800 10 50 100 200 300 400 500

Number of States Number of invariants Number of Traces

Model Size

invariants States

Detection time

• The duration

required for analysing the writing

• f a file is constant

(<1s)

• Exception of a too

small model size, as it refuses all traces (refusing a trace is more time consuming than accepting it)

CentraleSupelec 24

1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 100 200 300 400 500

Number of traces learned

Average Detection time (ms)

False positive rate

• False positive Rate

lowers with the number of the traces used to learn the model

• The more the model

is complete, the less it generates false positives

CentraleSupelec 25

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 50 100 150 200 250 300 350 400 450 500

False Positive Rate Number of traces learned

False Positive Rate

Conclusion

• Inferring a model of the control flow of a distributed application is

possible without relying on a global clock

• The originality of the approach relies
• On the generalization of the model
• The use of a dual model (automaton/temporal invariants)
• The detection produces an acceptable false positive rate
• It scales to large number of nodes
• Currently playing real attacks on XtreemFS

CentraleSupelec 26