SLIDE 1
Orchestrating Proxysql with OCP
Avi Apelbaum, Wix
linkedin/aviapelbaum github.com/wix aviap@wix.com
Hi.
I am Eyal DBA Team Leader @WIx.com
Orchestrating Proxysql with OCP Avi Apelbaum, Wix aviap@wix.com - - PowerPoint PPT Presentation
Orchestrating Proxysql with OCP Avi Apelbaum, Wix aviap@wix.com - - PowerPoint PPT Presentation
Orchestrating Proxysql with OCP Avi Apelbaum, Wix aviap@wix.com linkedin/aviapelbaum github.com/wix Hi. I am Eyal DBA Team Leader @WIx.com Wix in a Minute Our main production DBs Version 5.6.x & 5.7.x 3.2.12 3.0.9 DB instances
SLIDE 2
SLIDE 3
Wix in a Minute
SLIDE 4
Version 5.6.x & 5.7.x 3.2.12 3.0.9 DB instances ~574/30TB ~174/1.7TB ~43/Few GB
Most of them are multiple instances on a single physical server All DB’s are replicated with at least 2 DC’s replicas
Our main production DBs
SLIDE 5 Google
Our DB Clusters
Austin
Main masters DC
Tampa
Secondary masters DC
Amazon
In 2 availability zones
Amsterdam
Used now for read-only dev access
Austin & Tampa DCs have two instances for HA Most instances have dedicated BI slaves in Austin
SLIDE 6
Hi.
I am Avi DBA & DevOPS @WIx.com
SLIDE 7
AGENDA
Standard Solutions for Standard Problems Network Split Architecture Case OCP
SLIDE 8
Standard Solutions for Standard Problems
01
SLIDE 9
How we manage ~600 mysql nodes
SLIDE 10
How we manage ~600 mysql nodes
SLIDE 11
Orchestrator
github.com/github/orchestrator
SLIDE 12
How we manage failover of ~600 machines
SLIDE 13
How we manage failover of ~600 machines
SLIDE 14
Orchestrator
github.com/github/orchestrator
ProxySQL
www.proxysql.com
SLIDE 15
Orchestrator
github.com/github/orchestrator
ProxySQL
www.proxysql.com
Knows your MySQL topology Handles failover
SLIDE 16
Orchestrator
github.com/github/orchestrator
ProxySQL
www.proxysql.com
Knows your MySQL topology Handles failover
What happens here? Master-Slave changes
SLIDE 17
Orchestrator
github.com/github/orchestrator
ProxySQL
www.proxysql.com
Knows your MySQL topology Handles failover
Someone wakes up in the middle of the night Master-Slave changes Manual configuration
SLIDE 18
~1.5 min failover time
Service malfunctioning
Someone wakes up in the middle of the night Master-Slave changes Manual configuration
SLIDE 19
Network Split Architecture Case
02
SLIDE 20
DC 2 DC 1 Master Slave Slave ProxySQL Orchestrator Master Slave Slave Orchestrator ProxySQL
Replication of data
between 2 DCs
SLIDE 21
DC 2 DC 1 Master Slave Slave ProxySQL Orchestrator Master Slave Slave Orchestrator ProxySQL
RW RO
In case of a network split
we have to stop reads to one of the DCs
SLIDE 22
In case of architecture change
we have to notify/update all proxysql servers
DC 1 DC 1 Master Slave Slave ProxySQL Orchestrator Master Slave Slave Orchestrator Slave
SLIDE 23
Orchestrator
github.com/github/orchestrator
ProxySQL
www.proxysql.com
Knows your MySQL topology Handles failover
Consul Template
github.com/hashicorp/consul-template
Populates values from Consul into the file system.
SLIDE 24
Orchestrator
github.com/github/orchestrator
ProxySQL
www.proxysql.com
Knows your MySQL topology Handles failover
Consul Template
github.com/hashicorp/consul-template
Populates values from Consul into the file system.
OCP script
Gets Orchestrator to update a change in topology
SLIDE 25
Orchestrator
github.com/github/orchestrator
ProxySQL
www.proxysql.com
Knows your MySQL topology Handles failover
Consul Template
github.com/hashicorp/consul-template
Populates values from Consul into the file system.
OCP script
Gets ProxySQL to sample for a change...
C
- n
s u l T e m p l
...and in case
- f change -
download new topology
SLIDE 26
Orchestrator-Consule-ProxySQL (OCP)
Demo
SLIDE 27
SLIDE 28
OCP Benefits
03
SLIDE 29
Before
Application Servers
Connected directly to DB clusters Connected to an abstraction layer
After
SLIDE 30
Before
Application Servers Local Health Check
Connected directly to DB clusters Connected to an abstraction layer Upon failure, app servers wait for resume Upon failure, app servers can still work (bcp)
After
SLIDE 31
Before
Application Servers Local Health Check Configuration Change
Connected directly to DB clusters Connected to an abstraction layer
After
Manually Automatically Upon failure, app servers wait for resume Upon failure, app servers can still work (bcp)
SLIDE 32
Before
Application Servers Local Health Check Configuration Change DB Failover
After
~1.5 minutes ~0.5 second Connected directly to DB clusters Connected to an abstraction layer Manually Automatically Upon failure, app servers wait for resume Upon failure, app servers can still work (bcp)
SLIDE 33
https://github.com/wix/ocp
Getting Started
SLIDE 34
github.com/github/orchestrator www.proxysql.com
Knows your MySQL topology Handles failover
Consul Template
github.com/hashicorp/consul-template
Populates values from Consul into the file system.
OCP script OCP script
Orchestrator ProxySQL
NEW
ProxySQL Cluster
SLIDE 35
2 options to get auto update
- n DB failures and network split
OCP
Connects 3 existing solutions (Orchestrator, Consul template & ProxySQL)
NEW
ProxySQL Cluster OCP script OCP script
+ Orchestrator
DB recovery time? Less moving parts
SLIDE 36
2 options to get auto update
- n DB failures and network split
OCP
Connects 3 existing solutions (Orchestrator, Consul template & ProxySQL)
What’s your topology? How can you make it update automatically? How much DB recovery time will be saved?
NEW
ProxySQL Cluster OCP script OCP script
+ Orchestrator
DB recovery time? Less moving parts
SLIDE 37
THANK YOU!
Any Questions?
Lithuania Ukraine
Vilnius Kyiv Dnipro
Wix Engineering Locations
Israel
Tel-Aviv Be’er Sheva
aviap@wix.com