The Magic that is DMS Who is this Tim guy... Fox! 20 years in IT - - PowerPoint PPT Presentation

▶

Nov 08, 2022 148 likes •430 views

Database Migration Service Postgres to Aurora... in 372 steps Timothy R. Fox Not a DBA The Magic that is DMS Who is this Tim guy... Fox! 20 years in IT Worked at Companies from 14 to 2,400 employees Unix/Linux Admin by Trade

SLIDE 1

The Magic that is DMS

Database Migration Service Postgres to Aurora... in 372 steps Timothy R. Fox Not a DBA

SLIDE 2

Who is this Tim guy...

Fox!
20 years in IT

‒ Worked at Companies from 14 to 2,400 employees ‒ Unix/Linux Admin by Trade ‒ Started with SunOS (not Solaris)

Now Head of Cloud Managed Services for CorpInfo

‒ Prior VP, Tech Ops for a $500M Public Company

PostgreSQL DBA (not Extraordinaire)

‒ I’m the reason you don’t raise your hand when someone asks if you know something

SLIDE 3

Hiked the JMT this Past Summer

Tuolumne Meadows

in Yosemite, to Mount Whitney

200+ Miles
20 Days…
But enough about me

SLIDE 4

SLIDE 5

What were we trying to do?

Fix stuff…

2 TB Postgres DB
Running on RDS
Doing 20-25,000

Transactions a second

Multiple outages

lasting 2-3 minutes each.

SLIDE 6

What stuff we were a fix’n

RDS runs on EBS
Reduce IO

Dependency on EBS

Known issues (now

corrected) with IO “hiccups” on snapshots

n Postgres
Recovery/Build time for

Read Replicas really high

SLIDE 7

Looked at Aurora

SLIDE 8

How to Get there

Run AWS Schema

Conversion Tool ‒ Define Schema changes and optionally apply those changes to a new Empty Aurora DB

Use DMS to move the data
Apply the remaining schema
bjects
Flip

SLIDE 9

What the heck is this DMS thing anyway?!?

As provided by AWS Sales

SLIDE 10

AWS Database Migration Service

Start your first migration in 10 minutes or less Keep your apps running during the migration Replicate within, to or from Amazon EC2 or RDS Move data to the same or different database engine

SLIDE 11

Customer Premises Application Users AWS

Internet

VPN

Start a replication instance
Connect to source and target databases
Select tables, schemas or databases

AWS DATABASE MIGRATION SERVICE

Let the AWS Database Migration Service create tables, load data and keep them in sync Switch applications over to the target at your convenience

AWS Database Migration Service

SLIDE 12

HOW DOES DMS WORK?

Provisions a replication server
Define source and target database endpoints
Create a task to migrate data from source to target
Existing data is copied from source tables to tables on the target
Changes on source are captured while the tables are loaded
Once load is complete, buffered changes are applied to the target
Additional changes captured on the source are applied to the target

until the task stopped or terminated

SLIDE 13

Replication Instance Source Target

START FULL LOAD

SLIDE 14

LOAD IS TABLE BY TABLE

Creates tables at target database
Sets up metadata required at target
Populates data from source
Each process loads one entire table
Can use multiple processes
Can be paused

‒ When restarted will continue from where it was stopped ‒ Will reload any tables that were currently in progress Replication Instance Source Target

SLIDE 15

DMS – CHANGE DATA CAPTURE (CDC)

“No Touch” design

‒ Reads recovery log of source database ‒ Using the engine’s native change data capture API ‒ No agent required on the source

Some requirements

‒ Oracle: Supplemental logging required ‒ MySQL: Full image row level bin logging required ‒ SQL Server: Recovery model bulk logged or full ‒ Postgres: wal_level = logical; max_replication_slots >= 1; max_wal_Senders >=1; wal_sender_timeout = 0

Changes captured and applied as units of single committed transactions
Activated when load starts
No changes are applied until load completes, then applied as soon as possible in

near real-time

SLIDE 16

CHANGES ARE TRANSACTIONAL & COME FROM THE LOGS

Migration Server Source Target

Update

t1 t2 t1 t2

SLIDE 17

CUTOVER – SHUT DOWN APPS & APPLY REMAINING CHANGES

Source Target Replication Instance

Update

SLIDE 18

FLIP!

Source Target Replication Instance

Update

SLIDE 19

AWS SCHEMA CONVERSION TOOL

Assessment Report

SCT analyses the source database and provides a

report with a recommended target engine and information on automatic and manual conversions Code Browser and recommendation engine

Highlights places that require manual edits and provides

architectural and design guidelines.

SLIDE 20

SLIDE 21

OTHER DMS MAGIC

NOT PART OF THE DEMO …

SLIDE 22

Replication Instance

Source Target

MULTIPLE TARGETS

Target Target

SLIDE 23

Migration Server

Source Target

MULTIPLE SOURCES

Source Source

SLIDE 24

Migration Server

Source Target

MULTIPLE SOURCES & TARGETS

Source Source Target

SLIDE 25

HOMOGENOUS OR HETEROGENEOUS

Replication Instance SQL Server MySQL Replication Instance Oracle Oracle Replication Instance Oracle Aurora

SLIDE 26

Lets See it in Action

SLIDE 27