SLIDE 1
Introduction
- ptimal strategy to defend network infrastructure
no standard for benchmarking
Network Defence Strategy Evaluation Page 2 / 17
NETWORK DEFENCE STRATEGY EVALUATION: SIMULATION VS. LIVE NETWORK - - PowerPoint PPT Presentation
NETWORK DEFENCE STRATEGY EVALUATION: SIMULATION VS. LIVE NETWORK - - PowerPoint PPT Presentation
NETWORK DEFENCE STRATEGY EVALUATION: SIMULATION VS. LIVE NETWORK Tuesday 9 th May, 2017 Jana Medkov Martin Husk, Martin Draar Introduction optimal strategy to defend network infrastructure no standard for benchmarking Network Defence
SLIDE 2
SLIDE 3
Introduction
- ptimal strategy to defend network infrastructure
no standard for benchmarking current state of strategy evaluation:
verification of the strategy’s decision logic evaluation in a simulated environment
simulated attacks replayed attacks
evaluation in a real environment, in-house attacks
Network Defence Strategy Evaluation Page 2 / 17
SLIDE 4
Research Questions
- 1. What are the differences between defence strategy evaluation in
simulated and real environments?
- 2. Does the attacker change his behaviour based on the defender’s
actions?
Network Defence Strategy Evaluation Page 3 / 17
SLIDE 5
Experiment Setup I
Semi-real Run during the experiment, the strategy was set to defend a network
- f honeypots in Masaryk University network
Simulation Run attacks observed on the network of honeypots before the experiment were replayed against the strategy
Network Defence Strategy Evaluation Page 4 / 17
SLIDE 6
Experiment Setup II
Honeynet Topology central logging mechanisms and a database of authentication attempts gate that had the capability to manipulate the traffic experiment setup described in demo session
x.x.x.2 test x.x.x.6 db x.x.x.4 www x.x.x.3
- x.x.x.5
mail attacker gateway honeypots strategy.py framework
Network Defence Strategy Evaluation Page 5 / 17
SLIDE 7
Experiment Setup III
Defence Requirements the service should not be compromised (attack success penalty), the service should be available (unavailability penalty), the firewall should not be reconfigured frequently (reconfiguration penalty).
Network Defence Strategy Evaluation Page 6 / 17
SLIDE 8
Tested Strategies I
Game Theory Based Strategy both the attacker’s and defender’s goals Nash equilibria to find the optimal defender’s strategy finite, non-zero, two player game in an extensive form Cost Sensitive Strategy considers the immediate defender’s cost associated with action action cost consists of
negative impacts: cost of reconfiguration, cost of unavailability positive impacts: potential damage that was mitigated by the defensive action
Network Defence Strategy Evaluation Page 7 / 17
SLIDE 9
Collected Data
experiment:
644 attacks, July and August 2016
historical data:
15,214 attacks, December 2011 till June 2016
Strategy Game theory Cost sensitive # attacks 207 437 # reconfigurations 2,374 1,029 # minutes blocked 5,294 22,467 # successful attacks 55 62
Network Defence Strategy Evaluation Page 8 / 17
SLIDE 10
Simulated and Semi-real Execution
There is a statistically significant difference between the evaluation results in a simulated environment and a semi-real environment. Environment Strategy Mean strategy score Stdev Semi-real Game-theory 803 1,279 Cost sensitive 489 938 Simulated Game-theory 1,006 2,371 Cost sensitive 1,109 2,343
Network Defence Strategy Evaluation Page 9 / 17
SLIDE 11
Attacker’s Behaviour
Attack Length
50 100 150 200 0.00 0.01 0.02 0.03 0.04 Minutes Density Cost sensitive Game theory None
Network Defence Strategy Evaluation Page 10 / 17
SLIDE 12
Attacker’s Behaviour
Correlation Between Attack Length and Strategy Result Environment Strategy Correlation 95% CI Semi-real Game-theory 0.11 [-0.02, 0.25] Cost sensitive 0.06 [-0.03, 0.15] Simulated Game-theory 0.35 [0.33, 0.36] Cost sensitive 0.41 [0.39, 0.42]
Network Defence Strategy Evaluation Page 11 / 17
SLIDE 13
Attacker’s Behaviour
Return Rate
Cost sensitive Game theory None 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Return rate
Network Defence Strategy Evaluation Page 12 / 17
SLIDE 14
Attacker’s Behaviour
Summary The attackers reacted to the defence as follows: the attacks had longer duration they returned more often to continue in the attack the strategy result is less dependent on the length of the attack
Network Defence Strategy Evaluation Page 13 / 17
SLIDE 15
Conclusion I
Lessons Learned the formal definition of requirements is not sufficient computational complexity of the strategies is often not reflected in the evaluation and have to be considered deployment in a real environment forces to address all aspects
- f the strategy
Network Defence Strategy Evaluation Page 14 / 17
SLIDE 16
Conclusion II
Summary the most common evaluation is executed in simulated environment using replayed or simulated attacks we show that the evaluation using replayed attacks is not sufficient, since the attackers change in behaviour affects the evaluation results we found several changes in attacker behaviour due to the network defence
Network Defence Strategy Evaluation Page 15 / 17
SLIDE 17
Conclusion III
Future Work we need a better, standardized methods for evaluation to enable objective comparison the evaluation should begin with simple, easily setup scenarios and continue to more realistic scenarios at least some of the evaluations should face real attackers
Network Defence Strategy Evaluation Page 16 / 17
SLIDE 18