Upgrade to MongoDB 4.0 Antonios Giannopoulos DBA @ - - PowerPoint PPT Presentation

Upgrade to MongoDB 4.0

Antonios Giannopoulos DBA @ Rackspace/ObjectRocket linkedin.com/in/antonis/

1

Introduction

www.objectrocket.com

2

Antonios Giannopoulos

Overview

• Upgrade Procedure
• Application Layer
• Middleware
• Database Layer
• Rollback Procedure
• Why 4.0?

www.objectrocket.com

3

MongoDB 4.0

www.objectrocket.com

4

MongoDB 4.0 released on June 26,2018 Current minor version is 4.0.3 You can obtain it from:

• Mongo Inc Download Center
• Percona Server Download Center
• Repos like yum, apt-get…

MongoDB 4.0

www.objectrocket.com

5

Upgrade Replica-Set

www.objectrocket.com

6

Upgrade the secondary, one at a time

• Shut down the mongod instance
• Replace the binary with the 4.0 binary
• Restart the member

Connect a mongo shell to the primary

• Issue rs.stepDown()
• Ensure a new Primary is elected

Upgrade the ex-Primary

• Shut down the mongod instance
• Replace the binary with the 4.0 binary
• Restart the member

Connect a mongo shell to the primary Enable backwards-incompatible 4.0 features db.adminCommand( { setFeatureCompatibilityVersion: "4.0" }

Sharded Cluster

www.objectrocket.com

7 s1 s2

Stop Balancer

• sh.stopBalancer()
• sh.getBalancerState()

Upgrade config servers

• Use the replica-set steps

Upgrade the shards

• Use the replica-set steps

Upgrade the mongos

• One at a time
• Replace binary and restart

Enable backwards-incompatible 4.0 features db.adminCommand( { setFeatureCompatibilityVersion: "4.0" } ) Restart the mongos

Application

• Upgrade Driver
• Upgrade Language
• Inspect the code base

www.objectrocket.com

8

Driver

www.objectrocket.com

9

Your driver must be compactible with 4.0

Driver

www.objectrocket.com

10

Driver

www.objectrocket.com

11

So just upgrade the driver?

www.objectrocket.com

12

The underling language version must also be compactible with your driver

Mongo 4.0 Compatible

Upgrade the driver/language

www.objectrocket.com

13

Perform safe upgrades /always have rollback in mind Python supports “virtual environments” Python “virtualenv” creates an isolated environment for Python projects. Each project can have its own dependencies, regardless of what dependencies every other project has. There are no limits to the number of environments ,they’re just directories containing scripts

Upgrade the driver/language

www.objectrocket.com

14

Upgrade the driver/language

www.objectrocket.com

15

What about other languages: There are equivalents of “virtualenv” like JAVA use classpath Ruby use Ruby Version Manager (RVM)

Upgrade the driver/language

www.objectrocket.com

16

Do not forget frameworks and their dependencies

• Very popular in PHP
• They have their own compatibility matrix

Should I always use the latest driver?

• Check the change log
• Check the open bugs

Driver version? Dunno

www.objectrocket.com

17

Inspect the logs Manipulate the logs to export driver name & version Quick hack:

less /var/log/messages |grep "driver: { name" | awk '{print $16, $17, $18, $19}' | uniq

Driver version? Dunno

www.objectrocket.com

18

Inspect the system.profile Collect a representative sample On a sharded cluster, all shards must examined.

Inspect the codebase

www.objectrocket.com

19

Each version Removes & Deprecates operators. Remove: Operator is no longer available Deprecate: Operator will be removed to the next version Take Actions:

• Patched the codebase against the removed operators
• Plan to replace the deprecated operators

Inspect the codebase

www.objectrocket.com

20

Removed Operators:

• $isolated
• perator:

If you have an existing partial index that includes the $isolated operator or a view that includes a $isolated operator, recreate the index

• r view without the operator in the definition before upgrading.

Deprecated Operators:

• maxScan
• geoNear command
• copydb and the clone commands

Inspect the codebase

www.objectrocket.com

21

Exception handling

• Error descriptions may change between versions
• Error format may also change
• New error codes may introduced
• New warning codes may introduced

Middleware

• Monitoring
• Backups
• Deployments
• Utilities
• OS changes
• TLS/SSL

www.objectrocket.com

22

Monitoring

www.objectrocket.com

23

The monitoring system must be able to connect to MongoDB Vast majority relies on db.serverStatus() A newer version may be necessary to installed prior to upgrade

Deployment & Managment

www.objectrocket.com

24

Automatic deployment scripts also connect to MongoDB For example:

• Deploy a replica-set (rs.add(), rs.status())
• Add a shard (sh.addShard())

A change to error code formats may affect deployment scripts User-roles may also change Management tools may also affected

Backup

www.objectrocket.com

25

• Filesystems Snapshots
• Copy files
• Hot Backup (Percona Server)
• Mongodump

They should all work but testing is recommended. In the case of restore, a downgrade may be required.

OS

www.objectrocket.com

26

Removes support for:

• SLES 11
• Ubuntu 12.04
• Debian 7

Deprecates:

• Windows 7/2008R2
• Windows 8/2012
• Windows 8.1/2012R2
• Ubuntu 14.04

Compatibility matrix: https://bit.ly/2ys8Crs

TLS

www.objectrocket.com

27

Removes support for TLS 1.0

• On systems where TLS 1.1+ is available

Latest PCI compliance standards require the use of TLS 1.1+

If you need to support TLS 1.0 (hopefully temporarily):

• Set to none either net.ssl.disabledProtocols or --sslDisabledProtocols

Test:

• penssl s_client -connect <host>:<port> -tls1 (tls1_1 and tls1_2)

Database Layer

• Configuration files
• Prerequisites
• Deprecated items
• Storage engine
• Miscellaneous
• FCV

www.objectrocket.com

28

mongo.conf

www.objectrocket.com

29

net.transportLayer cant be set to legacy anymore net.host: Resolves localhost IP address as configured instead of assuming 127.0.0.1 Can’t combine storage.journal.enabled: false with WiredTiger storage engine Can’t combine storage.indexBuildRetry with replication.replSetName When ssl.allowInvalidCertificates: true with x.509 authentication, an invalid certificate is only sufficient to establish a TLS/SSL connection but is insufficient for authentication.

Prerequisites

www.objectrocket.com

30

In order to upgrade featureCompatibilityVersion must be set to 3.6 How to check:

• db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )

How to set FCV to 3.6:

• db.adminCommand( { setFeatureCompatibilityVersion: <version> } )

,where <version> ”3.6” It automatically enables 3.6 new features

Prerequisites

www.objectrocket.com

31

3.6 new features

• UUID for collections
• $jsonSchema document validation
• Change Streams
• Chunk aware secondaries
• View definitions, document validators, and partial index filters that use 3.6 query features
• Sessions and retryable writes
• Users and roles with authenticationRestrictions

Sessions:

• Creates a system.session collection
• On sharded clusters is sharded on {_id:1}
• Has a 30 minute TTL index
• Pre-3.6.7 bug, wasn’t creating TTL and sharding
• May trigger a performance overhead (updates/deletes)

Prerequisites

www.objectrocket.com

32

In order to upgrade pv must be set to pv1 How to check:

• pv=rs.conf().protocolVersion

How to set pv1:

• Login to every replica-set and execute

cfg = rs.conf(); cfg.protocolVersion=1; rs.reconfig(cfg);

Enjoy the benefits of pv1

Prerequisites

www.objectrocket.com

33

Master-slave replication, no longer supported You must upgrade to a replica set Prior the upgrade Conversion involves downtime & affects high availability

• Stop mongo
• Start Master with --replSet <setname>
• Initialize the replica set

rs.initiate( { _id: "<setname>", members: [ { _id: 0, host: "<host:port>" } ] } )

• Add nodes, rs.add("<host:port>" )

Prerequisites

www.objectrocket.com

34

Removes MongoDB Challenge-Response (MONGODB-CR) auth mechanism Deprecated since 3.0 version. Only possible if you upgraded from 2.6 How to check:

db.getSiblingDB('admin').system.users.find({"credentials.MONGODB-CR":{$exists:true}})

How to upgrade to SCRAM:

db.adminCommand({authSchemaUpgrade: 1});

Considerations

• authMechanism on connection string
• Local users on sharded clusters

Storage engine

www.objectrocket.com

35

MongoDB 4.0 deprecates MMAPv1 It’s likely the next version to support only WiredTiger You should consider a switch to WiredTiger Change to WiredTiger may be challenging:

• In-place Updates
• Range Queries
• Different HW specs
• Different shard keys

Storage engine

www.objectrocket.com

36

storage: dbPath: <data dir> engine: mmapv1 mmapv1: <mmap configuration> storage: dbPath: <data dir> engine: wiredTiger mmapv1: <mmap configuration> wiredTiger: collectionConfig: blockCompressor: <value> engineConfig: cacheSizeGB: <value> directoryForIndexes: true journalCompressor: <value> indexConfig: prefixCompression: <value>

Storage engine

www.objectrocket.com

37

Switch to WT (without secondary reads) 1)Convert one secondary

• Alter the configuration
• Stop mongo, wipe data dir,start mongodb
• Initial sync converts the secondary to WT

2)Promote the secondary to become Primary

• rs.freeze(300) the MMAPv1 secondary
• rs.stepdown(300)

3)Burn Period

• Convert the remaining nodes
• Rollback to MMAPv1

Freeze

Storage engine

www.objectrocket.com

38

Switch to WT (secondary reads) 1)Convert one secondary 3)Promote the secondary to become Primary 2)Burn Period (Secondary) 4)Burn Period (Primary) Important: Make sure less secondary can serve your workload

Storage engine

www.objectrocket.com

39

What can I do in advance?

• Benchmark with a real workload
• Prepare to “upgrade” your hardware
• Divide and conquer databases/collections
• Optimize range scans, if possible
• Replace updates with inserts, if possible
• Change shard keys, if possible

Workloads that MMAPv1 may perform better

• Read-Only
• Updates that change a small fraction of the document (counter)

Miscellaneous

www.objectrocket.com

40

• Removes the limit on the amount of data that can be rolled back, was 300MiB
• The rollback time limit defaults to 1 day, was 30 minutes
• Configurable via rollbackTimeLimitSecs, wasn’t configurable
• If you don’t have enough oplog, rollback will fail
• Make sure oplog duration > rollbackTimeLimitSecs
• Disable Rollbacks createRollbackDataFiles
• The oplog can grow past its configured size limit

Miscellaneous

www.objectrocket.com

41

taskExecutorPoolSize: Number of Task Executor connection pools

• New default is 1, was 0.
• Value of 0 means
• number of cores < 4, the number of pools is 4.
• 4 <= number of cores =< 64, the number of pools is the number of cores.
• number of cores> 64, the number of pools is 64.

AsyncRequestsSenderUseBaton: Default true Enables performance optimization on Linux for scatter/gather operations

• n mongos when using a single Task Executor connection pool.

FCV

www.objectrocket.com

42

• To enable 4.0 new features you must execute:

db.adminCommand({setFeatureCompatibilityVersion: ”4.0"})

• Its advisable to wait for a brief period of time
• During this period none of the 4.0 features will be available
• Rollback to 3.6 would be more difficult after raise the FCV
• We will examine rollback in the next chapter

FCV – What does it do?

www.objectrocket.com

43

Enables:

• SCRAM-SHA-256
• New type conversion operators and enhancements (typical for major version)
• Multi-document transactions
• $dateToString option changes (onNull)
• New change stream methods
• Change stream resume token data type change (hex-encoded string)

Confidence level

www.objectrocket.com

44

Major Version Replica-set Sharded Cluster Reason 2.2 2.2.3 2.4 2.4.3 2.4.6 Chunk Migration 2.6 2.6.6 2.6.6 Optimizer 3.0 3.0.5 3.0.8 Data loss 3.2 3.2.6 3.2.11 ASIO bug 3.4 3.4.4 3.4.7 Minor bugs 3.6 3.6.3 3.6.8 sessions

Downgrade

• Downgrade FCV
• Rollback a replica set
• Rollback a sharded cluster

www.objectrocket.com

45

Downgrade to 3.6

www.objectrocket.com

46

Downgrade FCV to 3.6

• db.adminCommand({setFeatureCompatibilityVersion: "3.6"})

Remove incompatible features:

• Drop views, document validators, and partial index filters that use 4.0 query features
• Downgrade the authentication mechanism from SCRAM-SHA-2 to "SCRAM-SHA-1"

db.updateUser( ”username", { mechanisms: [ "SCRAM-SHA-1" ], pwd: <newpwd> } )

• Take a backup
• Verify that all replica set members are in sync
• Use latest 3.6 if possible

Downgrade Replica-Set

www.objectrocket.com

47

Downgrade the secondary, one at a time

• Shut down the mongod instance
• Replace the binary with the 3.6 binary
• Restart the member

Connect a mongo shell to the primary

• Issue rs.stepDown()
• Ensure a new Primary is elected

Upgrade the ex-Primary

• Shut down the mongod instance
• Replace the binary with the 3.6 binary
• Restart the member

Sharded Cluster

www.objectrocket.com

s1 s2

Stop Balancer

• sh.stopBalancer()
• sh.getBalancerState()

Downgrade config servers

• Use the replica-set steps

Downgrade the shards

• Use the replica-set steps

Downgrade the mongos

• One at a time
• Replace binary and restart

Start Balancer

• sh.startBalancer()
• sh.getBalancerState()

Why 4.0?

• Transactions
• Secure authentication
• Change streams
• Non blocking Secondary reads
• Performance
• Miscellaneous improvements

www.objectrocket.com

49

Transactions

www.objectrocket.com

50

MongoDB 4.0 supports multi-document transactions

• Only available on replica sets
• Requires FCV = “4.0”
• Requires 4.0 compatible drivers
• Requires WiredTiger storage engine
• Transactions buffer lives in cache
• 16MB document size limit due to the oplog

Transactions

www.objectrocket.com

51

SCRAM-SHA-256

www.objectrocket.com

52

• SHA1 has been deprecated due to its security vulnerabilities
• SHA2 is the successor of SHA1 and SHA-256 its one of its variants
• SHA-256 produces a 256 bits hash (vs 160 on SHA1)
• SHA2 is less vulnerable to collision attacks
• SHA2 is vulnerable to collision attacks length extension attack

Change Streams

www.objectrocket.com

53

• Open a change stream cursor for a single database*
• Open a change stream cursor for a deployment**
• Adds the startAtOperationTime option
• Resume token is now a hex-encoded string. Allows comparison and sort.

*excluding admin, local, and config database ** In 3.6 the type is BinData

Secondary Reads

www.objectrocket.com

54

Before 4.0

• Writes are applied as batches.
• Block reads during batches to avoid a "wrong" order.
• Periodically the readers have to wait for replication batches to be applied
• Batches needs a lock that requires all reads to complete before it can be taken
• Write heavy workloads increase latency

In 4.0

• Timestamps in the storage engine
• Using transactions to get a consistent view of data at a specific "cluster time".
• Secondary reads takes advantage of snapshots isolation
• Relax the replication lock/allows reads while writes are happening.
• Index format changed to avoid collisions (unique indexes)

WiredTiger

www.objectrocket.com

55

• WiredTiger 3.1.1: (July 12, 2018) vs WiredTiger 2.9.2: (December 23, 2016)
• Improved Eviction (WT-3683, WT-3437, WT-4141)
• Improved Checkpoints (WT-4111)
• Reduces the number of internal writes by 50%
• WiredTiger timestamps (uses MongoDB timestamps)

Miscellaneous

www.objectrocket.com

56

$near and $nearSphere supports querying on sharded collections Adds key option for the $geoNear aggregation operator and geoNear command Mongos can log slow statements: Command line options: slowms Configuration file options: operationProfiling.slowOpThresholdMs Mongos can rate limit slow statements: Command line options: --slowOpSampleRate Configuration file options: operationProfiling.slowOpSampleRate Mongo shell method supports convertShardKeyToHashed

Miscellaneous

www.objectrocket.com

57

Aggregation framework supports $convert Converts a value to specified type:

Questions?

www.objectrocket.com

58

www.objectrocket.com

59

We’re Hiring!

Looking to join a dynamic & innovative team? https://www.objectrocket.com/careers/

• r email careers@objectrocket.com

Thank you!

Address: 401 Congress Ave Suite 1950 Austin, TX 78701 Support: 1-800-961-4454 Sales: 1-888-440-3242 www.objectrocket.com

60