Co Conc ncer erto to: Cooperative Network-Wide Telemetry with - - PowerPoint PPT Presentation

▶
co conc ncer erto to cooperative network wide telemetry
SMART_READER_LITE
LIVE PREVIEW

Co Conc ncer erto to: Cooperative Network-Wide Telemetry with - - PowerPoint PPT Presentation

Oct 29, 2023 346 likes •528 views

Co Conc ncer erto to: Cooperative Network-Wide Telemetry with Controllable Error Rate Yiran Li Kevin Gao Xin Jin Wei Xu Net etwork rk Tel elem emet etry ry Provide ides s Us Usef eful ul Status tus Knowle wledge dge Query

slide-1
SLIDE 1

Co Conc ncer erto to: Cooperative Network-Wide Telemetry with Controllable Error Rate

Yiran Li Kevin Gao Xin Jin Wei Xu

slide-2
SLIDE 2

Net etwork rk Tel elem emet etry ry Provide ides s Us Usef eful ul Status tus Knowle wledge dge

2

Query Interface

Telemetry System Core

Switch Config Telemetry Tuples

Switch DDoS Det. New TCP Port Scan

Expressive & High Fidelity Operators Operates Packets

  • f Whole Network

in Real-Time

slide-3
SLIDE 3

Traffic Traffic Executing at SP: √General Processing × Scalability Problem Executing at PISA Switch √Real-Time Processing × Limited Stages & Memory

Exe xecuti uting ng Lo Location ion: : Stream eam Proces essor sor vs.

  • s. PISA

A Swit itch ch

3

Protocol Independent Switch Architecture

Parser Deparser Stage 1

  • Mem. ALU

Stage 2

  • Mem. ALU

DDoS Det. Network Telemetry System

Task Assignment SP vs. PISA Switch

slide-4
SLIDE 4

Di Differe rent nt Swi witche ches s Pla lay y Di Differe rent nt Role les

4

Cluster Manager

Driver Program

Worker Executor Executor Executor Executor Worker Executor Executor Executor Executor Worker Executor Executor Executor Executor

Big Data Frameworks

Dynamically Assign to Available Executor Identical Workers, Executors

S1 S2 S3

SP

Query Switches with Different Roles Statically Assign to Specific Switch

Telemetry Systems

vs.

Core

slide-5
SLIDE 5

Usi sing g Swi witche ches s In Inde depe pendent dently ly Is Is In Insu suffic fficient ient

5

S1 S2 S3

SRC Edge DST Edge

SP

SRC

1 packetStream 2 .map (p => (p.ip.sip ,p.ip. dip )) 3 .distinct (( sip , dip ) => (sip , dip )) 4 .map ((_, dip ) => (dip ,1)) 5 .scan (( dip ,_) => dip , sum ) 6 .filter (( dip , count ) => count ==T) 7 .map (( dip , count ) => dip ) 1 packetStream 2 .map (p => (p.ip.sip ,p.ip. dip )) 3 .distinct (( sip , dip ) => (sip , dip )) 4 .map ((_, dip ) => (dip ,1)) 5 .scan (( dip ,_) => dip , sum ) 6 .filter (( dip , count ) => count ==T) 7 .map (( dip , count ) => dip )

DDoS Detection Query Unused Unused Splitting Query Between SP & Edge Switch Dynamically

slide-6
SLIDE 6

Usi sing g Swi witche ches s In Inde depe pendent dently ly Is Is In Insu suffic fficient ient (Cont.) t.)

6

S1 S2 S3

SRC Edge DST Edge

SP

SRC

1 packetStream 2 .map (p => (p.ip.sip ,p.ip. dip )) 3 .distinct (( sip , dip ) => (sip , dip )) 4 .map ((_, dip ) => (dip ,1)) 5 .scan (( dip ,_) => dip , sum ) 6 .filter (( dip , count ) => count ==T) 7 .map (( dip , count ) => dip )

Static Splitting & Using Switches Independently Duplicated Unused Wasted

1 packetStream 2 .map (p => (p.ip.sip ,p.ip. dip )) 3 .distinct (( sip , dip ) => (sip , dip )) 4 .map ((_, dip ) => (dip ,1)) 5 .scan (( dip ,_) => dip , sum ) 6 .filter (( dip , count ) => count ==T) 7 .map (( dip , count ) => dip )

DDoS Detection Query

slide-7
SLIDE 7

Concert erto:

  • : Cooper

erative ative Net etwork rk-Wide Wide Te Telem emet etry ry

  • Challenge
  • Splitting queries among switches while meeting resource & network constraints
  • Cooperative query execution model
  • Splitting query to multiple PISA switches
  • Each switch processes tuples locally
  • Various operations on different switches
  • Best-effort tuple processing
  • Automatic query placement
  • Analyzing query restrictions from AST
  • Formulating query placement as MIP
  • Result
  • Reduce the stream processor’s workload by up to 19 ×
  • Achieve 104 × lower error rate with the same workload

7 SP Config

Stream Processor

Query

Concerto Core Q

1

Q

3

Q

2

Switch Config Intermediate Tuples Result

Switch

slide-8
SLIDE 8

Coope perativ ative Query y Executi ution

  • n Mo

Mode del

8

S1

1 packetStream 2 .map (p => (p.ip.sip ,p.ip. dip )) 3 .distinct (( sip , dip ) => (sip , dip )) 4 .map ((_, dip ) => (dip ,1)) 5 .scan (( dip ,_) => dip , sum ) 6 .filter (( dip , count ) => count ==T) 7 .map (( dip , count ) => dip )

1→3 3→5 5→7 S2 S3

Phase 7 Phase 4

SP 4→7

5 .scan (( dip ,_) => dip , sum ) 6 .filter (( dip , count ) => count ==T) 7 .map (( dip , count ) => dip )

slide-9
SLIDE 9

Query y Executi ution

  • n on Swi

witche hes

9

S1 S2

Stage 1 06 07 05 05 05 05 04 Parser 07 05 Deparser PHV PHV Pkt Pkt

Four-Stage PISA Switch 07:map 06:filter 05:scan Subquery 1: Subquery 2:

  • Mem. ALU

Stage

Stage 2 Stage 3 Stage 4 Like Bloom Filter Only Based on Phase

slide-10
SLIDE 10

Concerto rto Puts s Mo More Op Operati tions

  • ns on S

Swi witche ches

  • Switch hardware
  • 4 stages
  • 0.5 Mb of registers at each stage
  • Results
  • Stateless filtering: 2.1 × 106
  • Independent stateful: 1.4 × 106
  • Concerto: 86

10

1 packetStream 2 .map (p => (p.ip.sip ,p.ip. dip )) 3 .distinct (( sip , dip ) => (sip , dip )) 4 .map ((_, dip ) => (dip ,1)) 5 .scan (( dip ,_) => dip , sum ) 6 .filter (( dip , count ) => count ==T) 7 .map (( dip , count ) => dip )

f1: S1 → S6 S1

2333

S2

2 - - -

S6

5556

S7

5556

S3

3333

S5

4 - - -

f2: S2 → S6 f3: S2 → S7

Stream Processor

7

Result Tuples

S4

4 - - -

Flow # Tuples # Stages t1, t2 t3, t4 t5 t6, t7 d3 d5 f1 442628 50034 1033 25 3 3 f2 1383594 113584 1739 36 4 3 f3 307941 8874 2194 25 3 3 f1+f2 1826222 163618 2772 61 5 3 f2+f3 1691535 122458 3933 61 4 4 f1+f2+f3 2134163 172492 4966 86 5 4

slide-11
SLIDE 11

Eval aluat uatio ion n Setup up

  • Questions: workload reduction, error rate guarantee, scalability
  • Topology
  • CAIDA trace: captured at a backbone ISP link from New York to San Paulo
  • Compared systems
  • Stateless: Everflow, DREAM
  • EdgeAll: Sonata
  • AnyAggre: OpenSketch, UnivMon, Marple
  • Metric: # tuples to the stream processor (same as Sonata)

11

Topology # Sites # Links Claranet 15 018 ATT North America 25 056 Cesnet-10 52 063 OTEGlobe 93 103 ATT North America

slide-12
SLIDE 12

Concerto Reduces SP’s Workload on Various Queries

12

SSpreader New TCP Port Scan DDoS In. Flows SSH Brute Slowloris 108 106 104 102 #Tuples

Single-Query Performance on Various Queries

Stateless EdgeAll AnyAggre Concerto

Better Log- Scaled

slide-13
SLIDE 13

Concerto rto Ac Achie ieve ves s Mu Much Lowe wer Error Rate

13

Error Rate Requirement

107 106 105 104

#Tuples

101 10-1 10-3 10-5

Error Rate (%) One Query

108 107 106 105 104 101 100 10-1

Error Rate (%) Four Queries Stateless EdgeAll AnyAggre Concerto 104

slide-14
SLIDE 14

Concerto rto Scal ales s Well ll

14

One Query

104 102 100

Normalized #Tuples

Normalized Workloads on Various Topologies

Four Queries

104 102 100

Two Queries

103 100

Stateless EdgeAll AnyAggre Concerto

Similar Similar Similar Different Different Normalized To Concerto

slide-15
SLIDE 15

Conclusi lusion

  • n
  • We propose a cooperative query execution model
  • Mimics network routing, each switch processes tuples locally
  • Independent of the underlying routing method
  • Applies to arbitrary topology
  • We provide a method to automatically compile queries to PISA switches
  • Analyzes the query placement requirement from AST
  • Formulates and optimizes query placement on switches using MIP
  • We show that the cooperative query execution of Concerto is effective
  • Reduces the stream processor’s workload by as much as 19 times
  • Achieves an error rate of 104 times lower than state-of-the-art systems

15

slide-16
SLIDE 16

Th Thanks! anks! Q& Q&A