Microservices reativos usando a stack do Netfmix na AWS Diego - - PowerPoint PPT Presentation

▶

Oct 18, 2023 27 likes •609 views

Microservices reativos usando a stack do Netfmix na AWS Diego Pacheco Principal Software Architect at ilegra.com @diego_pacheco www.ilegra.com NetfmixOSS Stack Why Netfmix? Netfmix My Problem Billions Requests Per Day Social

SLIDE 1

Microservices reativos usando a stack do Netfmix na AWS

Diego Pacheco Principal Software Architect at ilegra.com @diego_pacheco

SLIDE 2

SLIDE 3

www.ilegra.com

SLIDE 4

NetfmixOSS Stack

SLIDE 5

Why Netfmix?  Billions Requests Per Day

 1/3 US internet bandwidth  ~10k EC2 Instances  Multi-Region  100s Microservices  Innovation + Solid Service  SOA, Microservices and DevOps Benchmark

 Social Product  Social Network  Video  Docs  Apps  Chat Scalability Distributed T eams Could reach some Web Scale Netfmix My Problem

SLIDE 6

AWS

SLIDE 7

Cloud Native

SLIDE 8

Principles  Stateless Services  Ephemeral Instances  Everything fails all the time  Auto Scaling / Down Scaling  Multi AZ and multi Region  No SPOF  Design for Failure (expected)  SOA  Microservices  No Central Database  NoSQL  Lightweight Serializable Objects  Latency tolerant protocols  DevOps Enabler

 Immutable Infrastructure  Anti-Fragility

SLIDE 9

Right Set of Assumptons

SLIDE 10

Microservices

SLIDE 11

Reactive

SLIDE 12

Java Drivers X REST

X

SLIDE 13

Simple View of the Architecture

Zuul UI Microservice Cassandra Cluster

SLIDE 14

Stack

SLIDE 15

OSS

SLIDE 16

Zuul

SLIDE 17

Zuul

SLIDE 18

Karyon: Microbiology - Nucleus

SLIDE 19

 Reactive Extensions + Netty Server  Lower Latency under Heavy Load  Fewer Locks, Fewer Thread Migrations  Consumes Less CPU  Lower Object Allocation Rate

RxNetty

SLIDE 20

Karyon: CODE

SLIDE 21

Karyon: Reactive

SLIDE 22

Karyon: Reactive

SLIDE 23

Eureka and Service Discovery http://microservices.io/patterns/server-side-discovery.html

SLIDE 24

Eureka

 AWS Service Registry for Mid-tier  Load balancing and Failover  REST based  Karyon and Ribbon Integration

SLIDE 25

Eureka

SLIDE 26

Eureka and Service Discovery

SLIDE 27

Availability

SLIDE 28

Histryx

SLIDE 29

 IPC Library  Client Side Load Balancing  Multi-Protocol (HTTP, TCP, UDP)  Caching*  Batching  Reactive

Ribbon

SLIDE 30

Ribbon CODE

SLIDE 31

Ribbon CODE

SLIDE 32

 Reactive Extension of the JVM  Async/Event based programming  Observer Pattern  Less 1mb  Heavy usage by Netfmix OSS Stack

RX-Java

SLIDE 33

Archaius

 Confjguration Management Solution  Dynamic and T yped Properties  High Throughtput and Thread Safety  Callbacks: Notifjcations of confjg changes  JMX Beans  Dynamic Confjg Sources: File, Db, DynamoDB, Zookeper  Based on Apache Commons Confjguration

SLIDE 34

Archaius + Git Microservice Microservice Slave Side Car Central Internal GIT Property Files File System Microservice Microservice Slave Side Car File System Microservice Microservice Slave Side Car File System

SLIDE 35

Asgard

SLIDE 36

Asgard

SLIDE 37

Packer

JOB Create

Bake/Provision

Launch

Deploys

SLIDE 38

Dynomite: Distributed Cache https://github.com/Netfmix/dynomite

SLIDE 39

Dynomite  Implements the Amazon Dynamo Similar to Cassandra, Riak and DynamoDB Strong Consistency – Quorum-like – No Data Loss Pluggable Scalable Redis / Memcached Multi-Clients with Dyno Can use most of redis commands Integrated with Eureka via Prana

SLIDE 40

 Isolate Failure – Avoid cascading  Redundancy – NO SPOF  Auto-Scaling  Fault T

lerance and Isolation

 Recovery  Fallbacks and Degraded Experience  Protect Customer from failures – Don’t throw Failures -> Failures VS Errors

Dynomite: Distributed Cache

SLIDE 41

Dynomite: Internals

SLIDE 42

Oregon D1 Oregon D2 N California D3 Eureka Server Eureka Server

Prana Prana

Prana

Multi-Region Cluster

SLIDE 43

Dynomite: CODE

SLIDE 44

Dynomite Contributions

https://github.com/Netfmix/dynomite https://github.com/Netflix/dynomite/pull/207 https://github.com/Netflix/dynomite/pull/200

SLIDE 45

Caos Engineering

SLIDE 46

Gatling

Stress T esting T

Scala DSL Run on top of Akka Simple to use

SLIDE 47

Chaos Arch

Zuul Microservice N1 Microservice N2 Cassandra Cluster Zuul Eureka ELB

SLIDE 48

Running…

SLIDE 49

Chaos Results and Learnings

 Retry confjguration and Timeouts in Ribbon  Right Class in Zuul 1.x (default retry only SocketException)

 RequestSpecifjcRetryHandler (Httpclient Exceptions)  zuul.client.ribbon.MaxAutoRetries=1  zuul.client.ribbon.MaxAutoRetriesNextServer=1  zuul.client.ribbon.OkT

RetryOnAllOperations=true

Eureka Timeouts  It Works  Everything needs to have redudancy  ASG is your friend :-)  Stateless Service FTW

SLIDE 50

Microservice Producer Kafka / Storm :: Event System

SLIDE 51

Chaos Results and Learnings

Before:

 Data was not in Elastic Search  Producers was loosing data

After:

 No Data Loss  It Works

Changes:

 No logging on Microservice :( (Log was added)  Code that publish events on a try-catch  Retry confjg in kafka producer from 0 to 5

SLIDE 52

SLIDE 53

Main Challenges

SLIDE 54

Hacker Mindset

SLIDE 55

SLIDE 56

Next Steps  IPC  Spinnaker  Containers  Client side Aggregation  DevOps 2.0 -> Remediation / Skynet

SLIDE 57

Pocs

https://github.com/diegopacheco/netflixoss-pocs http://diego-pacheco.blogspot.com.br/search/label/netflix?max-results=30

SLIDE 58

Microservices reativos usando a stack do Netfmix na AWS

Diego Pacheco Principal Software Architect at ilegra.com @diego_pacheco

www.ilegra.com

NetfmixOSS Stack

Why Netfmix?  Billions Requests Per Day

 1/3 US internet bandwidth  ~10k EC2 Instances  Multi-Region  100s Microservices  Innovation + Solid Service  SOA, Microservices and DevOps Benchmark

 Social Product  Social Network  Video  Docs  Apps  Chat Scalability Distributed T eams Could reach some Web Scale Netfmix My Problem

Cloud Native

Right Set of Assumptons

Microservices

Reactive

Java Drivers X REST

X

Simple View of the Architecture

Stack

 Reactive Extensions + Netty Server  Lower Latency under Heavy Load  Fewer Locks, Fewer Thread Migrations  Consumes Less CPU  Lower Object Allocation Rate

 AWS Service Registry for Mid-tier  Load balancing and Failover  REST based  Karyon and Ribbon Integration

 IPC Library  Client Side Load Balancing  Multi-Protocol (HTTP, TCP, UDP)  Caching*  Batching  Reactive

 Reactive Extension of the JVM  Async/Event based programming  Observer Pattern  Less 1mb  Heavy usage by Netfmix OSS Stack

 Confjguration Management Solution  Dynamic and T yped Properties  High Throughtput and Thread Safety  Callbacks: Notifjcations of confjg changes  JMX Beans  Dynamic Confjg Sources: File, Db, DynamoDB, Zookeper  Based on Apache Commons Confjguration

Dynomite  Implements the Amazon Dynamo Similar to Cassandra, Riak and DynamoDB Strong Consistency – Quorum-like – No Data Loss Pluggable Scalable Redis / Memcached Multi-Clients with Dyno Can use most of redis commands Integrated with Eureka via Prana

 Isolate Failure – Avoid cascading  Redundancy – NO SPOF  Auto-Scaling  Fault T

 Recovery  Fallbacks and Degraded Experience  Protect Customer from failures – Don’t throw Failures -> Failures VS Errors

Multi-Region Cluster

Dynomite Contributions

Stress T esting T

Scala DSL Run on top of Akka Simple to use

Chaos Arch

 Retry confjguration and Timeouts in Ribbon  Right Class in Zuul 1.x (default retry only SocketException)

Eureka Timeouts  It Works  Everything needs to have redudancy  ASG is your friend :-)  Stateless Service FTW

Before:

After:

Changes:

Next Steps  IPC  Spinnaker  Containers  Client side Aggregation  DevOps 2.0 -> Remediation / Skynet

Pocs

Microservices reativos usando a stack do Netfmix na AWS

Diego Pacheco Principal Software Architect at ilegra.com @diego_pacheco

Obrigado!