Backwaters: Security Streaming Platform Comcast TPX Security - - PowerPoint PPT Presentation

▶
backwaters security streaming platform
SMART_READER_LITE
LIVE PREVIEW

Backwaters: Security Streaming Platform Comcast TPX Security - - PowerPoint PPT Presentation

Jun 21, 2023 9 likes •180 views

Backwaters: Security Streaming Platform Comcast TPX Security Solutions Engineering (SSE) The Team Chris Maenner Ryan Van Antwerp Will Weber Principal Security Developer Principal Security Developer Senior Security Developer 2 Agenda

slide-1
SLIDE 1

Backwaters: Security Streaming Platform

Comcast TPX Security Solutions Engineering (SSE)

slide-2
SLIDE 2

2

The Team

Chris Maenner

Principal Security Developer

Ryan Van Antwerp

Principal Security Developer

Will Weber

Senior Security Developer

slide-3
SLIDE 3

3

Agenda

  • Apache Kafka Overview
  • Intelligence Driven Security
  • Methods of Receiving Logs
  • Architecture of Backwaters: Security Streaming Platform
  • How security utilizes Apache Kafka's API
slide-4
SLIDE 4

4

Apache Kafka Overview

Apache Kafka is a distributed streaming platform which has three key capabilities:

  • Publish and subscribe to streams of records, similar to a message queue
  • Store streams of records in a fault-tolerant durable way
  • Process streams of records as they occur

Kafka is generally used for two broad classes of applications:

  • Building real-time streaming data pipelines that reliably get data between systems or

applications

  • Building real-time streaming applications that transform or react to the streams of data

Kafka includes four core data-centric APIs:

  • Producer
  • Consumer
  • Streams
  • Connector

Reference: https://kafka.apache.org/documentation/#gettingStarted

slide-5
SLIDE 5

5

Intelligence Driven Security

LEVEL 1

NETWORK & AUTHENTICATION TOOLS & METADATA APPLICATION DATA

LOG PRODUCERS

THREAT CORRELATION ADVANCED THREAT DETECTION DATA SCIENCE

SIEM

SECURITY EVENTS CORRELATION AND ALERTING

TOOLS

CLOUD & DATA CENTER INFRASTRUCTURE

ADVANCED DETECTION

NETWORK, TOOLS, INFRASTRUCTURE

REAL-TIME KNOWN THREAT CORRELATION BEHAVIORAL ANALYTICS, THREAT SIMULATION & DECEPTION UNKNOWN THREAT DETECTION

LEVEL 2 LEVEL 3

TRANSFORM LAYER

BACKWATERS KAFKA

LEVEL 1

slide-6
SLIDE 6

6

Methods of Receiving Logs

Comcast Data Centers

Options:

  • Kafka Producer
  • Syslog Producer

Amazon Web Services Cloud

Options:

  • EC2 Producer
  • Lambda Producer

Microsoft Azure Cloud

Options:

  • Azure Functions Producer
  • Azure VM Producer
slide-7
SLIDE 7

7

Syslog Ingest Path

Other sources Linux Servers

Data Producers

Comcast Private Cloud

Message Bus Consumers

Elastic Beats Elastic Logstash

Primary Secondary

Availability Zones

Load Balancer(s)

Central West East

Security Information and Event Management (SIEM) Data Science Tools Compliance Tools Log Indexing Kafka Compatible Tools

slide-8
SLIDE 8

8

Comcast Managed

Comcast Private Cloud

AWS Consumer Path

Kafka

AWS Cloud

Logstash Winlogbeat

AWS Direct Connect

AWS VPC

Amazon EC2 Amazon S3

slide-9
SLIDE 9

9

AWS Ingest Path

Comcast Private Cloud

Backwaters

SIEM

Data Science Tools Elastic Search Cluster

Comcast Managed

AWS Direct Connect

AWS Virtual Private Cloud (VPC)

AWS S3 AWS GuardDuty AWS Lambda

slide-10
SLIDE 10

10

Backwaters Multi-Tenant Data Framework

Servers Comcast Private Cloud Azure Event Hubs Azure Functions Azure Log Analytics

Kafka

AWS Direct Connect

Amazon Web Services Microsoft Azure Cloud

Amazon S3

EC2 VPC Subnet

Azure Express Route

slide-11
SLIDE 11

11

Apache Kafka’s API

  • The Producer API allows an application to publish a stream of records to one or more Kafka

topics

  • The Consumer API allows an application to subscribe to one or more topics and process the

stream of records produced to them

  • The Streams API allows an application to act as a stream processor, consuming an input stream

from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams

  • The Connector API allows building and running reusable producers or consumers that connect

Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table

  • The AdminClient API allows managing and inspecting topics, brokers, and other Kafka objects

Reference: https://kafka.apache.org/documentation/#api

slide-12
SLIDE 12

12

Apache Kafka Producer/Consumer API

The Producer API:

  • Write access to one or more topics
  • Allows applications to send streams of

data to topic(s)

Kafka Broker

Partition 2 Partition 1 Topic 2 Partition 2 Partition 1 Topic 1

Consumer Producer Producer Producer The Consumer API:

  • Read access to one or more topics
  • Read streams of data from topic(s)

Consumer

slide-13
SLIDE 13

13

Apache Kafka Streams API

  • High level abstraction language using Java’s API
  • Unbounded, continuous real-time flow of records
  • You don’t need to explicitly request new records, you just receive them
  • Domain Specific Language (DSL) is built on top of the Streams Processor API:
  • Built-in abstractions for streams and tables:
  • Kstream: append-only ledger (INSERT only)
  • Ktable: UPSERT changelog stream for one partition
  • GlobalKTable: UPSERT changelog stream for all partitions
  • Supports stateless and stateful transformations:
  • Map: unique keys to values
  • Filter: evaluate Boolean to retain or drop elements
  • Aggregations (e.g. count, reduce)
  • Joins (e.g. Inner, Left, Outer)
  • Windowing (e.g. group records that have the same key)

Reference: https://kafka.apache.org/documentation/streams

slide-14
SLIDE 14

14

Kafka Streams API (Transform)

Backwaters

Kafka Streams app transform object(s) and write to new topic Comcast Cloud

Source Raw Data

Producers Consumers

Raw Topic "2019-01-10 20:20:39"; \ "alice”; \ "Windows”; \ "Desktop”; \ "10.0.0.126" { "timestamp": "2019-01-10 20:20:39", "username": "alice", "os": "Windows", "systemType": "Desktop", "ipAddress": "10.0.0.126" } Parsed Topic

slide-15
SLIDE 15

15

Apache Kafka Connect API

  • Connectors tool for scalably and reliably streaming data between Kafka and other systems
  • Kafka Connect is intended to be run as a service
  • Kafka Connect currently supports two modes of execution:
  • Standalone: all work is performed in a single process (Simplest)
  • Distributed: handles automatic balancing of work, allows you to scale up (or down) dynamically
  • Core Concepts and APIs:
  • Connectors come in two flavors (e.g. Pull or Push):
  • SourceConnectors: import data (e.g. JDBCSourceConnector would import relational database)
  • SinkConnectors: export data (e.g. HDFSSinkConnector export topic to an HDFS file)
  • Connectors are responsible for breaking jobs into a set of Tasks:
  • SourceTask: pull interface with two APIs, commit and commitRecord
  • SinkTask: push interface
  • REST API Layer:
  • View the status/configuration of connectors
  • Alter current behavior (e.g. change config or restart task

Reference: https://kafka.apache.org/documentation/#connect

slide-16
SLIDE 16

16

Kafka Connect API (SourceConnectors)

Kafka Streams app transform object(s) and write to new topic

{ "timestamp": "2019-01-10 20:20:39", ”user": "alice", "os": "Windows", ”type": "Desktop", "ipAddress": "10.0.0.126" } Parsed Topic

Backwaters

Comcast Cloud

Consumers

Kafka Connect app performing JDBC connection to database

timestamp user

  • s

type ipAddress 2019 2019-01 01-10 20:20:39 10 20:20:39 Alice Alice OSX OSX Desktop Desktop 10.0.0.126 10.0.0.126

slide-17
SLIDE 17

Questions?