Containers @ Wrapp Powered by AWS EC2 Container Service Jude DSouza - - PowerPoint PPT Presentation
Containers @ Wrapp Powered by AWS EC2 Container Service Jude DSouza Solutions Architect @ Wrapp Phone: +46 767085740 Email: jude@wrapp.com About Me q Jude DSouza Stockholm, Sweden Karachi, Pakistan jude@wrapp.com q MS
Powered by AWS EC2 Container Service
Jude D’Souza Solutions Architect @ Wrapp Phone: +46 767085740 Email: jude@wrapp.com
q Jude D’Souza – Stockholm, Sweden ß Karachi, Pakistan jude@wrapp.com q MS – Distributed Systems KTH Royal Institute of Technology, Stockholm, Sweden q Solutions Architect @ Wrapp www.wrapp.com q Passion for building large scale distributed systems
Wrapp delivers personal offers and rewards based on where you shop, dine and the brands you follow. q Microservice Architecture q Infrastructure
q 13 ECS Clusters q 50+ Instances q 90+ Services
q Technology Stack
q Operating System - Ubuntu q Consul, Registrator, AWS(EC2, ECS, Lambda), Docker, HAProxy q Monitoring – Sensu, Riemann q Logs - Logentries and Sumo q Languages - Go, Python, Java
– Service Orchestration – Service Discovery – Service Deployments – Service Availability & Scalability – Container Registry
– Microservice Architecture @ Wrapp
Service Orchestration
q Via runlist configs On instance boot: q Determine autoscaling group q Pull runlist for autoscaling group q Run containers specified in config q Add these to supervisor misc
api
misc Autoscaling group
Autoscaling group api Autoscaling group
PULL RUNLIST CONFIGS
Sample Runlist configuration file
ü Service Orchestration Service Deployments Service Availability & Scalability Container Registry
Service Deployments q Find instances where service is running (via serf) q SSH into these instances q Do rolling deploy (Terminate container, Launch container with new version)
misc
api
misc Autoscaling group
Sample serf data for services SERF CLUSTER
Autoscaling group api Autoscaling group
ü Service Orchestration ü Service Deployments Service Availability & Scalability Container Registry
Service Availability
q Supervisord
Service Scalability
q Service itself can't scale q Scale only by scaling autoscaling group
misc
api
misc Autoscaling group
SERF CLUSTER
Autoscaling group api Autoscaling group
misc
Container Registry
q Hosting our own container registry q Backed by s3
ü Service Orchestration ü Service Deployments ü Service Availability & Scalability ü Container Registry
Service Orchestration q Runlists q Supervisor Service Deployments q Supervisor Service Availability q Supervisor Service Scalability
q EC2 Autoscaling
Container Registry
q Self-hosting
Service Discovery q Serf q HAProxy Service Discovery q Consul q Registrator q HAProxy
Service Orchestration Service Deployments Service Availability Service Scalability (with lambda) AWS Container Registry + Service stats (cpu + memory) + Service monitoring + Service Provisioning
PUBLIC SUBNET
PRIVATE SUBNET
Elastic Load Balancer
api api api
Microservices ZONE A ZONE B ZONE C api.wrapp.com DNS
Cluster Overview
Services Overview
Instances Overview
misc
misc Autoscaling group Scaling Amazon ECS Services Automatically Using Amazon CloudWatch and AWS Lambda: https://aws.amazon.com/blogs/compute/scaling-amazon-ecs-services-automatically-using-amazon-cloudwatch-and-aws-lambda/ CloudWatch Alarm SNS Topic AWS Lambda EC2 Container Service ECS Instances Trigger CloudWatch alarm from ECS Metrics (CPU/ Memory) Trigger Lambda function from SNS Topic +1 DesiredCount in Service
api
emails rewards auth notifications eventbus
Typical ECS Instance
ü Memory allocation by Service ü Allocate CPU Units by Service ü Efficient Resource utilization (Container placement)
Typical ECS Instance
q Logspout q Rsyslog q LogEntries
Logentries: https://logentries.com Logspout: https://github.com/gliderlabs/logspout
logspout rsyslog
Containers launched via ECS Log router for Docker Forwards logs to logentries
q Our Experience q Generally good service (ECS is oK) q Great support q We don’t need to host these services
q Always improving to make their services better and better
ü Supports built-in service discovery, loadbalancing and routing ü Weave Flux: Supports built-in service discovery, loadbalancing and routing ü Docker Swarm: Supports orchestrating containers ü Fleet - Container Orchestration
api
8080
Consul server-b Consul server-a Consul server-c
users
8082
auth
8081
users
8082
Registrator Registrator Registrator Consul Agent Consul Agent Consul Agent
Key/value store
Service catalog
10.0.0.1 10.0.0.2 10.0.0.3
HA Consul Cluster
Consul server-b Consul server-a Consul server-c
users
8083
Registrator Consul Agent
10.0.0.3
HAProxy auth
8082
Consul Template frontend users-in bind users:80 mode http default_backend users backend users server Server-a 10.0.0.1:8080 … server Server-c 10.0.0.3:8080 … #### frontend auth-in bind auth:80 mode http default_backend auth backend auth server Server-a 10.0.0.1:8081 … server Server-c 10.0.0.3:8081 …
>> curl users/v1/users/1 { "name": "jude" "gender": "male" }
HA Consul Cluster
DNS Hack (192.168.x.x)
Built-in (like kube-proxy, weave-proxy) ?
q Service Discovery q Service Routing q Service Load balancing Approach 1: ELB Per service? q Costly - 90+ services, 70+ ELBs Approach 2: Single ELB for all services? ü Service Routing works ü Service Load balancing works q No health check per service q No way to return HTTP 503 (but possible via haproxy http check option) Benefits: q Removes tools like Consul and Registrator
Elastic Load Balancer
8080 8081 8082
ECS Boto3: create_service()
Route53: ELB.dns = elb-services frontend users-in bind users:80 mode http default_backend users backend users server users elb-services:8080 … #### >> curl users/ping
SINGLE ELB FOR MULTIPLE SERVICES APPROACH
Ø Cron scheduling q Setup ecs tasks to run on cron schedule Ø ECS Event stream q Actionables on certain events (when deploy completes) Ø ECS Container Logging Ø Optionally Integrate ECS Cluster with EC2 Autoscaling Group
q The Wrapp Architecture before ECS q Transition to ECS q Service Discovery @ Wrapp with Consul, Registrator and HAProxy q Our perceived shortcomings from ECS q Built-in Service Discovery q Cron scheduling on Tasks q Event stream
Contact: Jude D’Souza Email: jude@wrapp.com Phone: +46767085740 Twitter: Wrapp Tech @wrapp_tech