Integrating a temporal component into multi-criteria majority-rule - - PowerPoint PPT Presentation
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Integrating a temporal component into multi-criteria majority-rule sorting models Application to the cyber-defense context Arthur VALKO Director :
Context MCDA strategy Time integration Experimentation Concluding remarks and future work
Integrating a temporal component into multi-criteria majority-rule sorting models
Application to the cyber-defense context Arthur VALKO Director : Patrick Meyer Supervisors : Alexandru-Liviu Olteanu, David Brosset November 22, 2018
Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 1 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work
Outline
1
Context Decision problem
2
MCDA strategy Operational constraints Model choice
3
Time integration Hierarchical model Learning Process
4
Experimentation Test platform Experimentation
5
Concluding remarks and future work
Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 2 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Decision problem
MCDA & cyber-defense : ship example
Actions Mission Decision maker Attacks Functionalities
Criteria Alternatives Preferences
Ship captain
Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 3 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Decision problem
Dashboard for the decision maker
Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 4 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Operational constraints
Operational constraints
The ship captain does not necessarily need a ranking of the actions, but rather a qualitative evaluation of each of them. → He / she wishes to have the final word! → sorting algorithm Evaluation scales of the criteria are heterogeneous and have a strong meaning for the decision maker. He / she (cyber defender) does not trust black boxes. → High readability of the decision recommendation required → outranking method
Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 5 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Model choice
MR-Sort
Sorting outranking model Various extensions possible to increase expressiveness Output easy to read and to explain Indirect learning process
Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 6 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Model choice
Further need
Consequences of actions might vary with time on the various criteria
T-4 T-3 T-2 T-1
T
T+1 T+2 T+3 T+4 T+5
Good evaluation Bad evaluation
5/5 4/5 4/5 5/5 5/5 99% 60% 30% 60% 35% T+1 T+2 T+3 T+4 T+5 5/5 5/5 5/5 3/5 5/5 15% 15% 99% 50% 99% T+1 T+2 T+3 T+4 T+5
Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 7 / 19 Context MCDA strategy Time integration Experimentation Concluding remarks and future work
Time integration
1
Context Decision problem
2
MCDA strategy Operational constraints Model choice
3
Time integration Hierarchical model Learning Process
4
Experimentation Test platform Experimentation
5
Concluding remarks and future work
Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 8 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work
Time integration
Multiple options : Increase the number of criteria (J · T) One "time"-criterion per time step Loss of readability for DMs The DM may have difficulties to understand the output preference model Time aggregation Loss of information (intra- and inter-criterion) Our proposal : hierarchical approach (H) Time structure conservation Better readability for DMs
Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 9 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Hierarchical model
Hierarchical model
Good evaluation Bad evaluation
T+1 T+2 T+3 T+4 T+5
1/5 3/5 5/5 5/5 5/5
30% 50% 99% 99% 99%
majority threshold : 5 criteria weights G B bad alternative good alternative separation profile majority threshold : 6 criteria weights G B bad alternative good alternative separation profile majority threshold : 7 criteria weights G B bad alternative good alternative separation profile majority threshold : 5 criteria weights G B bad alternative good alternative separation profile majority threshold : 5 criteria weights G B bad alternative good alternative separation profile G good alternative majority threshold : 6 criteria weights B bad alternative separation profile T+1 T+2 T+3 T+4 T+5Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 10 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Learning Process
Learning process
T-4 T-3 T-2 T-1
T
T+1 T+2 T+3 T+4 T+5
5/5 4/5 4/5 5/5 5/5 99% 60% 30% 60% 35%
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Learning Process
The MR-Sort hierarchical learning algorithm
Mixed-Integer Program : comparison with MR-SortJ·T Increase of number of profiles and number of majority thresholds Increase of number of variables (integer ones) Increase of number of constraints Study : Learning time as a function of problem size Inferred model quality as a function of problem size Cross-analysis between and MR-Sort with an increase of the number of criteria (MR-SortJ·T) and the hierarchical approach (MR-SortH).
Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 12 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Test platform
Test platform
Mixed integer program
T+1 T+2 T+3 T+4 T+5 0/5 0/5 2/5 4/5 5/5 45% 40% 30% 60% 75% T+1 T+2 T+3 T+4 T+5 5/5 5/5 5/5 3/5 5/5 15% 15% 99% 50% 99%fi fi fi
fi
fi fi fi fi fi fi fi fi fi fi fi fi fi fi fi fi fi fi Models generator Random MR-SortJ·T or MR-SortH models Select number of criteria (|J|), time steps (|T|) and categories (k) Alternatives generator Select number of criteria |J|, time steps |T| and the number of examples Assign alternatives with one of generated models Learn preference parameters via MR-SortJ·T and MR-SortH models Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 13 / 19 Context MCDA strategy Time integration Experimentation Concluding remarks and future work Experimentation
Experimentation
Parameters : Number of criteria : {3, 5, 7}, 3 time steps Number of categories k : {3, 5} Number of assignment examples : {10, 20, 30, 50, 70} Classification accuracy tests Comparison between inferred model and the original one with 10 000 new alternatives
Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 14 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Experimentation
Learning times
Global learning time on the set of test MR-SortH models are more difficult to learn (often > 1h) Important standard deviation MR-SortJ·T shorter learning time (0.2 - 10 seconds)
Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 15 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Experimentation
Classification accuracy with original model, MR-SortJ·T
Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 16 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Experimentation
Classification accuracy with original model, MR-SortH
Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 17 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work
Concluding remarks and future work
Hierarchical model : Long learning times Adds a time component into the decision-making process. Decision recommendation decomposition for better explanation. Good classification accuracy Future work : Apply the model to a real-world case Meta-heuristic learning method Automatic explanation of recommendations
Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 18 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work
Thank you for your attention. Any questions?
Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 19 / 19