Core Intel On the bank secret service Krzysztof Adamski # Mariusz - - PowerPoint PPT Presentation

▶
core intel
SMART_READER_LITE
LIVE PREVIEW

Core Intel On the bank secret service Krzysztof Adamski # Mariusz - - PowerPoint PPT Presentation

Oct 08, 2022 507 likes •1.14k views

Core Intel On the bank secret service Krzysztof Adamski # Mariusz Derela Miami 18th May 2017 Are security breaches common? https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/4

slide-1
SLIDE 1

Core Intel

Krzysztof Adamski # Mariusz Derela On the bank secret service

Miami 18th May 2017

slide-2
SLIDE 2

Are security breaches common?

https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/4 32412/bis-15-302-information_security_breaches_survey_2015-full-report.pdf

slide-3
SLIDE 3

Carbanak

3 https://securelist.com/blog/research/68732/the-great-bank-robbery-the-carbanak-apt/

slide-4
SLIDE 4

Core Intel is a part of ING Cyber Crime Resilience Programme to structurally improve the capabilities for the cybercrime

  • prevention
  • detection and the
  • response

CoreIntel

4

slide-5
SLIDE 5
  • Measures against e-banking fraud, DDoS and Advanced Persistent Threats (APTs).
  • Threat intelligence allow to respond to, or even prevent, a cybercrime attack
  • (This kind of intelligence is available via internal and external parties and includes both
  • pen and closed communities)
  • Monitoring, detection and response to “spear phishing”
  • Detection/mitigation of infected ING systems’
  • Baselining network traffic/anomaly detection
  • Response to incidents (knowledge, tools, IT environment)
  • Automated feeds, automated analysis and historical data analysis

The reasoning

5

slide-6
SLIDE 6

What is there on the market nowadays?

6

slide-7
SLIDE 7

The world is not enough

slide-8
SLIDE 8

So the challenge is…

slide-9
SLIDE 9

Market leaders Benelux Growth markets Commercial Banking Challengers

Most of our data is within Europe

9

slide-10
SLIDE 10

Market leaders Benelux Growth markets Commercial Banking Challengers

but we operate globally

10

slide-11
SLIDE 11

Expect the unexpected to collect all the data

11

slide-12
SLIDE 12
  • What kind of data do we need?
  • Where is our data located?
  • How we can potentially capture it?
  • What are the legal implications?

So there is a challenge to capture „all” the data

12

slide-13
SLIDE 13

Core Intel architecture

slide-14
SLIDE 14

So what you would like to see is…

Photo credit: edgarpierce via Foter.com / CC BY

slide-15
SLIDE 15

…In fact it is slightly more complicated

slide-16
SLIDE 16

All has its own purpose. Let’s see in details.

16 Photo credit: https://www.pexels.com/photo/dslr-camera-equipments-147462/

slide-17
SLIDE 17

Local data collector

17

slide-18
SLIDE 18

But tell how to capture that data

18 https://observer.viavisolutions.com/includes/popups/taps/tap-vs-span.php

slide-19
SLIDE 19

Broker settings: Replication factor >= 3 min.insync.replicas = 2 unclean.leader.election.enable = false replica.lag.time.max.ms Producer settings: acks = all retries = Integer.MAX_VALUE max.block.ms = Long.MAX_VALUE block.on.buffer.full = true To have data in order max.in.flight.requests.per.connection = 1

Kafka producer configuration (as we don’t like losing data)

19

Good overview here: https://www.slideshare.net/JayeshThakrar/kafka-68540012

slide-20
SLIDE 20

Central data collector

20

slide-21
SLIDE 21

Time is crucial here

21 Photo credit: Cargo Cult via Foter.com / CC BY

slide-22
SLIDE 22

But your business data more, so proceed with caution

22 Photo credit: https://www.pexels.com/photo/white-caution-cone-on-keyboard-211151/

slide-23
SLIDE 23
  • Network bandwidth control
  • quota.consumer.default
  • quota.producer.default

Kafka mirror maker configuration

23

slide-24
SLIDE 24

Secure data: listeners=SSL://host.name:port ssl.client.auth=required ssl.keystore.location ssl.keystore.password ssl.key.password ssl.truststore.location ssl.truststore.password

Kafka mirror maker configuration

24

Secure data in transit

slide-25
SLIDE 25

Streaming data

25

slide-26
SLIDE 26

spark.yarn.maxAppAttempts spark.yarn.am.attemptFailuresValidityInterval spark.yarn.max.executor.failures spark.yarn.executor.failuresValidityInterval spark.task.maxFailures spark.hadoop.fs.hdfs.impl.disable.cache spark.streaming.backpressure.enabled=true spark.streaming.kafka.maxRatePerPartition

Spark on yarn streaming configuration

26

slide-27
SLIDE 27

In memory data grid

27

val rddFromMap = sc.fromHazelcastMap("map-name-to-be-loaded")

slide-28
SLIDE 28

Let’s find something in these logs

28 Photo credit: https://www.flickr.com/photos/65363769@N08/12726065645/in/pool-555784@N20/

slide-29
SLIDE 29

Matching

29

Tornado - a Python web framework and asynchronous networking library - http://www.tornadoweb.org/ MessagePack – binary transport format http://msgpack.org/

slide-30
SLIDE 30
  • Automatically & continually match network logs <->threat intel
  • When new threat intel arrives, against full history network logs
  • When new network logs arrive, against full history threat intel
  • Alerts are shown in a hit dashboard
  • Dashboard is a web-based interfaces that provide flexible charts, querying, aggregation

and browsing

  • Quality/relevance of an alert is subject to the quality of IoC feeds and completeness of

internal log data.

Hit, alerts and dashboards

30

slide-31
SLIDE 31

Be smart with your tooling

31 Photo credit https://www.flickr.com/photos/12749546@N07/

slide-32
SLIDE 32

and leverage e.g. elasticsearch templates

32

slide-33
SLIDE 33

Data mapping:

  • doc_value
  • fielddata
  • fields

Cluster settings to check: gateway.recover_after_nodes gateway.recover_after_master_nodes gateway.recover_after_data_nodes indices.recovery.max_bytes_per_sec indices.breaker.total.limit indices.breaker.fielddata.limit

Elasticsearch configuration

33

slide-34
SLIDE 34

For those who know how to use heavy equipment

34 Photo credit: News Collection & Public Distribution @techpearce2 via Foter.com / CC BY

slide-35
SLIDE 35

Long data storage - HDFS

35

slide-36
SLIDE 36

Kafka offset management

36

slide-37
SLIDE 37

Core Intel allows users to perform advanced analytics on network logs using a set of powerful tools

  • Spark API to write code to process large data sets on a cluster
  • perform complex aggregations to collect interesting statistics
  • run large scale clustering algorithms with Spark’s MLLib
  • run graph analyses on network logs using Spark’s GraphX
  • transform and extract data for use in another system (which are better for specific analytics or

visualization purposes)

  • Kafka, co you can write own Consumers and Producers to work with your data
  • to perform streaming analysis on your data
  • to implement your own alerting logic
  • Toolset
  • Programming languages: Scala, Java, Python
  • IDE’s: Eclipse / Scala IDE, IPython Notebook and R Studio

Advanced analytics

37

slide-38
SLIDE 38

How do we schedule the jobs

38

slide-39
SLIDE 39

How to keep everything under control

39 Photo credit: https://www.flickr.com/photos/martijn141

slide-40
SLIDE 40

Monitoring crucial points in your data pipeline

40

slide-41
SLIDE 41

Something for smart guys

41 Photo credit: https://www.flickr.com/photos/jdhancock/5173498203/

slide-42
SLIDE 42

Plenty of data to analyze

42

slide-43
SLIDE 43

Challenger on the operations side. Are containers the answer?

43

slide-44
SLIDE 44

OpenShift HA deployment

44 http://playbooks-rhtconsulting.rhcloud.com/playbooks/installation/installation.html

slide-45
SLIDE 45

OpenShift Architecture

45

OSE VXLAN RT - zone Openshift set of clusters BR0(OVS) VTEP Phisical Network (ISP ECF)

T1: VlanT1 T1: Nodes Affinity: T1 Anti Affinity: [O66|R41]

. . . Tenants namespaces BR0 (OVS) VTEP

T2: VlanT2 T2: Nodes Affinity: T2 Anti Affinity: [O66|R41] Tn: VlanTn Tn: Nodes Affinity: Tn Anti Affinity: [O66|R41]

OSE - masters Infra nodes T1 nodes T2 nodes Tn nodes InnerPodT1 10.1.1.2 InnerPodT2 10.1.2.2 T2 Project VNID: 302 T1 Project VNID: 301 Tn Project VNID: n InnerPodT3 10.1.3.2 T3 Project VNID: 303 ... ... InnerPodTn 10.1.n.2

slide-46
SLIDE 46

OpenShift – Ingestion Layer

46

slide-47
SLIDE 47

OpenShift – Ingestion Layer

47

+

slide-48
SLIDE 48

OpenShift – Ingestion Layer

48

slide-49
SLIDE 49

OpenShift – Ingestion Layer

49

slide-50
SLIDE 50

OpenShift – Ingestion Layer – Pros & Cons

50

  • Rolling Update
slide-51
SLIDE 51

OpenShift – Ingestion Layer – Pros & Cons

51

  • Rolling Update
  • Triggers
slide-52
SLIDE 52

OpenShift – Ingestion Layer – Pros & Cons

52

  • Rolling Update
  • Triggers
  • AutoScale
slide-53
SLIDE 53

OpenShift – Ingestion Layer – Pros & Cons

53

  • Rolling Update
  • Triggers
  • AutoScale
  • Healthchecks
slide-54
SLIDE 54

OpenShift – Elasticsearch Stack

54

slide-55
SLIDE 55

OpenShift – Challanges

55

  • Persistent Storage
  • Rack Awarness

http://dailypicksandflicks.com/2011/10/25/did-you-know-the-worlds-best-selling-toy/cat-with-rubiks-cube/

slide-56
SLIDE 56

OpenShift – „PetSet” (Stateful Services)

56

slide-57
SLIDE 57

OpenShift – Persistent Storage

57

slide-58
SLIDE 58

OpenShift – Rack Awarness

58

slide-59
SLIDE 59

OpenShift – Capacity

59

slide-60
SLIDE 60

Q&A

krzysztof.adamski@ingservicespolska.pl @adamskikrzysiek https://pl.linkedin.com/in/adamskikrzysztof mariusz.derela@ingservicespolska.pl @mariusz_derela https://www.linkedin.com/in/mariusz-derela-30649a69

slide-61
SLIDE 61

61