Storms in the Cloud Designing and using a fault injection system - - PowerPoint PPT Presentation

▶
storms in the cloud
SMART_READER_LITE
LIVE PREVIEW

Storms in the Cloud Designing and using a fault injection system - - PowerPoint PPT Presentation

Jun 11, 2023 421 likes •1.06k views

Storms in the Cloud Designing and using a fault injection system Michalis Zervos @mzervos http://venturebeat.com/ http://www.pcworld.com/ Google News - http://thenextweb.com/ https://twitter.com/netflixhelps Google News -

slide-1
SLIDE 1

Storms in the Cloud

Designing and using a fault injection system

Michalis Zervos @mzervos

slide-2
SLIDE 2

http://venturebeat.com/ http://www.pcworld.com/ Google News - http://thenextweb.com/ https://twitter.com/netflixhelps Google News - http://mashable.com/ http://www.itnews.com.au Google News - http://www.itnews.com.au

slide-3
SLIDE 3
slide-4
SLIDE 4

Service Resilience

  • Not a solved problem
  • Goal is:
  • 100% uptime
  • No degradation
  • Responsive

slide-5
SLIDE 5

Traditional testing

  • Unit tests
  • Functional / Integration
  • End to end

slide-6
SLIDE 6

Cloud services – Testing challenges

  • Continuous evolution
  • Multiple dependencies
  • Global distribution
  • Traffic fluctuation

slide-7
SLIDE 7

Cloud services – Fundamentals

  • Auto-scaling
  • Redundancy
  • Monitoring and detection systems
  • Auto-mitigation / Failover mechanisms
  • Staged deployments
  • Data replication

slide-8
SLIDE 8

The extra mile

  • Embrace failure
  • Break the system
  • Adjust the engineering process

slide-9
SLIDE 9

Storms in the Cloud

slide-10
SLIDE 10

Fault Injection System

Support diverse services Easy to use Verify resilience and behavior Simulate complex failures / real-life incidents

slide-11
SLIDE 11

Agenda

Designing a Fault Injection System Usage patterns

slide-12
SLIDE 12

Faults

  • Resource pressure
  • Network
  • Processes
  • Virtual machine
  • Platform
  • Application specific
  • Hardware
slide-13
SLIDE 13

Faults

  • Resource pressure
  • Network
  • Processes
  • Virtual machine
  • Platform
  • Application specific
  • Hardware
slide-14
SLIDE 14

Resource Pressure Faults

  • CPU
  • Memory
  • Physical
  • Virtual
  • Hard disk
  • Capacity
  • Read
  • Write

Available tools

  • consume.exe (Windows SDK)
  • stress (Unix)
  • Sysinternals tools
slide-15
SLIDE 15

Faults

  • Resource pressure
  • Network
  • Processes
  • Virtual machine
  • Platform
  • Application specific
  • Hardware
slide-16
SLIDE 16

Network faults

  • Layers
  • Transport (TCP/UDP)
  • Application layer (HTTP)
  • Types
  • Disconnect
  • Latency
  • Alter response codes (HTTP)
  • Packet reorder / loss (TCP/UDP)
  • Filters
  • Domain / IP / Subnet
  • URL path
  • Port / Protocol

Available tools

  • Network Emulator Toolkit (NEWT)
  • Fiddler core
slide-17
SLIDE 17

Faults

  • Resource pressure
  • Network
  • Processes
  • Virtual machine
  • Platform
  • Application specific
  • Hardware
slide-18
SLIDE 18

Process faults

  • Stop / Kill
  • Restart
  • Stop service
  • Start
  • Crash
  • Hang

Available tools

  • OS commands
  • Sysinternals tools
slide-19
SLIDE 19

Faults

  • Resource pressure
  • Network
  • Processes
  • Virtual machine
  • Platform
  • Application specific
  • Hardware
slide-20
SLIDE 20

Virtual Machine / OS faults

  • Stop
  • Restart
  • BSOD / Kernel panic
  • Change date
  • Re-image

Available tools

  • Cloud Management APIs
  • OS commands
  • Sysinternals tools
slide-21
SLIDE 21

Faults

  • Resource pressure
  • Network
  • Processes
  • Virtual machine
  • Platform
  • Application specific
  • Hardware
slide-22
SLIDE 22

Distributed platform faults

  • Quorum loss
  • Data loss
  • Move primary node
  • Remove replica

Available tools – Platform specific

  • Service Fabric testability APIs
slide-23
SLIDE 23

Faults

  • Resource pressure
  • Network
  • Processes
  • Virtual machine
  • Platform
  • Application specific
  • Hardware
slide-24
SLIDE 24

Application specific faults

  • Hooks
  • Instrument service code
  • Intercept / Re-route calls
  • No access to service code

Available tools

  • MSR Detours
  • TestApi – Managed Fault Injection
slide-25
SLIDE 25

Application specific faults

  • Hooks
  • Instrument service code
  • Intercept / Re-route calls
  • No access to service code

Available tools

  • MSR Detours
  • TestApi – Managed Fault Injection
slide-26
SLIDE 26

Faults

  • Resource pressure
  • Network
  • Processes
  • Virtual machine
  • Platform
  • Application specific
  • Hardware
slide-27
SLIDE 27

Hardware faults

  • Machine
  • Network devices
  • Rack
  • UPS
  • Datacenter
slide-28
SLIDE 28

Faults

  • Resource pressure
  • Network
  • Processes
  • Virtual machine
  • Platform
  • Application specific
  • Hardware
slide-29
SLIDE 29

Injection mechanism

  • VM External
  • VM Internal – Service code external  Agent
  • VM Internal – Service code internal  Hooks
slide-30
SLIDE 30

Injection mechanism

  • VM External
  • VM Internal – Service code external  Agent
  • VM Internal – Service code internal  Hooks
slide-31
SLIDE 31

External injection

  • VM / Region Stop
  • VM / Region Restart
  • Re-image

Target VM Target VM Cloud Management Service Cloud Management Service

slide-32
SLIDE 32

Injection mechanism

  • VM External
  • VM Internal – Service code external  Agent
  • VM Internal – Service code internal  Hooks
slide-33
SLIDE 33

VM internal injection - Agent

  • Resource pressure
  • Network
  • Processes
  • OS
  • Detours
  • …

Target Service VM Target Application

Virtual Machine

Operating System Fault Agent

slide-34
SLIDE 34

Injection mechanism

  • VM External
  • VM Internal – Service code external  Agent
  • VM Internal – Service code internal  Hooks
slide-35
SLIDE 35

VM internal injection - Hooks

  • Application behavior
  • Flexibility
  • Service specific

Target Application

slide-36
SLIDE 36

VM internal injection - Hooks

  • Application behavior
  • Flexibility
  • Service specific

Target Application

slide-37
SLIDE 37

Hooks Fault Agent VM External

slide-38
SLIDE 38

System Architecture

Target Service VMs

Fault Management Service Fault Agent Fault Agent Cloud Management Service Cloud Management Service

slide-39
SLIDE 39

System components

Auditing Automation Verification Reporting Security

slide-40
SLIDE 40

System components

Auditing Automation Verification Reporting Security

slide-41
SLIDE 41

Security and Safety

  • AuthN / AuthZ
  • Fault agents
  • Kill switch
  • Safety nets
slide-42
SLIDE 42

Security and Safety

  • AuthN / AuthZ
  • Fault agents
  • Kill switch
  • Safety nets
  • Integrate with Identity Provider

Azure Active Directory

  • Multi-Factor Authentication
  • Least-privilege principle
  • Granular access levels
slide-43
SLIDE 43

Security and Safety

  • AuthN / AuthZ
  • Fault agents
  • Kill switch
  • Safety nets
  • Secure communication – TLS/SSL
  • Code signing
  • Execution permissions
slide-44
SLIDE 44

Security and Safety

  • AuthN / AuthZ
  • Fault agents
  • Kill switch
  • Safety nets
slide-45
SLIDE 45

Security and Safety

  • AuthN / AuthZ
  • Fault agents
  • Kill switch
  • Safety nets

Auto fault removal

  • Agents – Service connectivity loss

Agent-side detection

  • Service malfunctioning

Auto-monitoring module

  • Unusual behavior

Anomaly detection

slide-46
SLIDE 46

System components

Auditing Automation Verification Reporting Security

slide-47
SLIDE 47

System components

Auditing Automation Verification Reporting Security

slide-48
SLIDE 48

Auditing

  • Faults
  • Fault agents
  • Management service
  • Clients
slide-49
SLIDE 49

System components

Auditing Automation Verification Reporting Security

slide-50
SLIDE 50

System components

Auditing Automation Verification Reporting Security

slide-51
SLIDE 51

Automation

  • Scheduling
  • Zero - configuration
  • Dependencies auto-discovery
slide-52
SLIDE 52

System components

Auditing Automation Verification Reporting Security

slide-53
SLIDE 53

System components

Auditing Automation Verification Reporting Security

slide-54
SLIDE 54

System components

Auditing Automation Verification Reporting Security

slide-55
SLIDE 55
slide-56
SLIDE 56
slide-57
SLIDE 57

Usage scenarios

  • Resilience verification
  • Test new features
  • Training
  • Verify staged deployments
  • Test detection, alerting, mitigation systems
  • Repro incidents
slide-58
SLIDE 58

Injection environment

Test Canary Production

slide-59
SLIDE 59

Recovery Games

slide-60
SLIDE 60

Recovery Games

Attacker

  • Inject faults
  • Provide hints

Defender

  • Assess
  • Analyze
  • Mitigate
slide-61
SLIDE 61

Recovery Games - Goals

  • Familiarize with monitoring tools
  • Recognize outage patterns
  • Train on assessing the impact
  • Root-cause / mitigation mindset
  • Practice log analysis
slide-62
SLIDE 62

In Invest t in in Fault In Inje jecti tion Testing

Resilience verification Test new features Training

Engineering process & culture

Michalis Zervos @mzervos

Core Hubs:
All Converters PDF Hub Video Hub Audio Hub Image Hub File Compressor