Protecting your OpenStack cloud with an automated backup and - - PowerPoint PPT Presentation

▶
protecting your openstack cloud with an automated backup
SMART_READER_LITE
LIVE PREVIEW

Protecting your OpenStack cloud with an automated backup and - - PowerPoint PPT Presentation

Aug 24, 2023 619 likes •903 views

Protecting your OpenStack cloud with an automated backup and recovery strategy Carlos Camacho Gonzalez Dan Macpherson Senior Software Engineer Principal Technical Writer Red Hat Red Hat November 14, 2018 Agenda Introduction Defining

slide-1
SLIDE 1

Protecting your OpenStack cloud with an automated backup and recovery strategy

Carlos Camacho Gonzalez Senior Software Engineer Red Hat November 14, 2018 Dan Macpherson Principal Technical Writer Red Hat

slide-2
SLIDE 2

INSERT DESIGNATOR, IF NEEDED 2

  • Introduction
  • Defining the strategy
  • Backup and Restoring the Undercloud
  • Backup and Restoring the Overcloud
  • Challenges and Ideas

Agenda

slide-3
SLIDE 3

INSERT DESIGNATOR, IF NEEDED 3

  • How did we meet?
  • Fast forward upgrades
  • Problems we’re trying to solve

Introduction

slide-4
SLIDE 4

INSERT DESIGNATOR, IF NEEDED 4

Protect against maintenance tasks failures (Undercloud, Overcloud control plane) Protect user space (Trilio, Freezer)

Backup categories

User workload Backend services Configuration files Log files Databases

slide-5
SLIDE 5

INSERT DESIGNATOR, IF NEEDED 5

Goal: Ensure you can restore the Undercloud and the Overcloud controllers no matter what… and all automatically!

slide-6
SLIDE 6

INSERT DESIGNATOR, IF NEEDED

Defining Backup Strategies for Individual Services

6

slide-7
SLIDE 7

INSERT DESIGNATOR, IF NEEDED 7

Database (Non-HA)

For example: backing up and restoring the undercloud. Backup:

  • Run the mysqldump command

Restore:

  • Create a new database
  • Start mariadb
  • Increase the packet size
  • Restore data from .sql files
slide-8
SLIDE 8

INSERT DESIGNATOR, IF NEEDED 8

Database (HA)

Backup:

  • Select an idle node
  • Backup the database
  • Backup the grants

Restore:

  • Disable VIP access to the database (iptables)
  • Stop Galera
  • Temporarily disable replication
  • Create a new database on each node
  • Set database permissions (root, clustercheck)
  • Synchronize the nodes
  • Enable replication
  • Start Galera
  • Import database and grants
  • Restore VIP access (iptables)
slide-9
SLIDE 9

INSERT DESIGNATOR, IF NEEDED 9

MongoDB

Used for Telemetry storage in Newton Backup:

  • mongodump
  • https://docs.mongodb.com/manual/reference/progr

am/mongodump/ Restore:

  • mongorestore
  • https://docs.mongodb.com/manual/reference/progr

am/mongorestore/

slide-10
SLIDE 10

INSERT DESIGNATOR, IF NEEDED 10

Redis

Used as an object store for services. TripleO overclouds use it for Telemetry object storage.

“Redis is very data backup friendly since you can copy RDB files while the database is running: the RDB is never modified once produced, and while it gets produced it uses a temporary name and is renamed into its final destination atomically using rename(2) only when the new snapshot is complete.”

  • https://redis.io/topics/persistence

Backup

  • Save the current state (redis-cli bgsave)
  • Copy the /var/lib/redis/dump.rdb

Restore

  • Stop Redis
  • Copy dump.rdb back to /var/lib/redis/
  • Start Redis
slide-11
SLIDE 11

INSERT DESIGNATOR, IF NEEDED 11

Pacemaker Configuration

Restore previous pacemaker configuration. Backup:

  • Config backup command (pcs config backup pacemaker_backup)
  • Creates an archive file with configuration

Restore:

  • Stop the cluster (pcs cluster stop --all)
  • Restore config (pcs config restore pacemaker_controller_backup.tar.bz2)
  • Start the cluster (pcs cluster start --all)
slide-12
SLIDE 12

INSERT DESIGNATOR, IF NEEDED 12

Swift

Swift object data as files. Usually part of a filesystem backup.

Backup:

  • Backup object files on each node (usually in /srv/node)
  • Don’t forget the xattrs (Swift object metadata)
  • Backup ringfiles and configuration (/etc/swift)

Restore:

  • Restore each node’s object files (usually to /srv/node)
  • Don’t forget the xattrs (Swift object metadata)
  • Restore ringfiles and configuration (/etc/swift)
  • Restart swift

Always important to include any xattrs option for rsync or tar commands: # tar --xattrs ... # rsync --xattrs ...

slide-13
SLIDE 13

INSERT DESIGNATOR, IF NEEDED 13

Backup relevant directories in your filesystem. You might need to restore a particular piece

  • f configuration at some point.

Recommended directories:

  • /etc/
  • /var/lib/<service>/ (e.g. glance, cinder, heat, etc)
  • kolla config (e.g. /var/lib/config-data)
  • /srv/node/ (don’t forget xattrs!)
  • /var/log/
  • /root/ (contains .my.cnf for root access to database)
  • Your cloud admin user directory. In TripleO:

○ /home/stack for the undercloud ○ /home/heat-admin for the overcloud

Filesystem Backup

slide-14
SLIDE 14

INSERT DESIGNATOR, IF NEEDED

Undercloud Backup and Restore

14

slide-15
SLIDE 15

INSERT DESIGNATOR, IF NEEDED 15

Backing up the Undercloud

Create snapshots. Virtual node? Backup the resources required to restore it back to a consistent state. Baremetal node? OpenStack < Queens Manual backups based

  • n either bash or

Ansible. OpenStack >= Queens TripleO CLI option “openstack undercloud backup”

slide-16
SLIDE 16

INSERT DESIGNATOR, IF NEEDED 16

Backing up the Undercloud

Manual steps:

  • penstack undercloud backup [--add-path ADD_FILES_TO_BACKUP] [--exclude-path

EXCLUDE_FILES_TO_BACKUP]

  • penstack undercloud backup --add-path /etc/ \
  • -add-path /var/log/ \
  • -add-path /root/ \
  • -add-path /var/lib/glance/ \
  • -add-path /var/lib/docker/ \
  • -add-path /var/lib/certmonger/ \
  • -add-path /var/lib/registry/ \
  • -add-path /srv/node/ \
  • -exclude-path /home/stack/
slide-17
SLIDE 17

INSERT DESIGNATOR, IF NEEDED 17

Backing up the Undercloud

CLI driven:

mysqldump --opt --single-transaction --all-databases > /root/undercloud-all-databases.sql sudo tar --xattrs --ignore-failed-read -cf \ UC-backup-`date +%F`.tar \ /root/undercloud-all-databases.sql \ /etc \ /var/log \ /root \ /var/lib/glance \ /var/lib/docker \ /var/lib/certmonger \ /var/lib/registry \ /srv/node \ /home/stack

slide-18
SLIDE 18

INSERT DESIGNATOR, IF NEEDED 18

Restoring the Undercloud

Strategy?

  • Restore the snapshot or nuke the node and install from scratch [1]

Reasons?

  • Transaction history might be hard to rollback after an upgrade
  • Single node no HA, easy to reinstall

How to do it?

  • Restore the configuration files
  • Restore the certificates files
  • Restore the databases
  • Run: `openstack undercloud install`

[1]:https://docs.openstack.org/tripleo-docs/latest/install/controlplane_backup_restore/03_undercloud_restore.html

slide-19
SLIDE 19

INSERT DESIGNATOR, IF NEEDED

Overcloud Backup and Restore

19

slide-20
SLIDE 20

INSERT DESIGNATOR, IF NEEDED 20

Overcloud Backup and Restore Strategy

  • Composable and agnostic automated backup and restore system
  • Ansible role - ansible-role-openstack-operations [1]
  • Foundational ansible tasks [2]

○ Allows you to set an external backup server and automatically configure it ○ Bootstrap node assignment ○ Ansible synchronize module (rsync wrapper) ○ Provides temporary SSH access to nodes ○ Tasks for database backup ○ Tasks for database restore (containerized HA) ○ Tasks to validate the database

  • Future goals:

○ More services (Pacemaker, Redis, Swift, etc) ○ Different backend architectures (Non-HA, non-containerized)

[1] http://git.openstack.org/cgit/openstack/ansible-role-openstack-operations/ [2] https://review.openstack.org/#/c/604439/

slide-21
SLIDE 21

INSERT DESIGNATOR, IF NEEDED

  • name: Initialize backup host

hosts: "{{ backup_hosts | default('backup') }}[0]" Tasks:

  • import_role:

name: ansible-role-openstack-operations tasks_from: initialize_backup_host

  • name: Backup MySQL database

hosts: "{{ target_hosts | default('mysql') }}[0]" vars: backup_server_hostgroup: "{{ backup_hosts | default('backup') }}" tasks:

  • import_role:

name: ansible-role-openstack-operations tasks_from: validate_galera

  • import_role:

name: ansible-role-openstack-operations tasks_from: enable_ssh

  • import_role:

name: ansible-role-openstack-operations tasks_from: backup_mysql

  • import_role:

name: ansible-role-openstack-operations tasks_from: disable_ssh 21

Backing up the overcloud

slide-22
SLIDE 22

INSERT DESIGNATOR, IF NEEDED

  • name: Initialize backup host

hosts: "{{ backup_hosts | default('backup') }}[0]" tasks:

  • import_role:

name: ansible-role-openstack-operations tasks_from: initialize_backup_host

  • name: Restore MySQL database on galera cluster

hosts: "{{ target_hosts | default('mysql') }}" vars: backup_server_hostgroup: "{{ backup_hosts | default('backup') }}" tasks:

  • import_role:

name: ansible-role-openstack-operations tasks_from: set_bootstrap

  • import_role:

name: ansible-role-openstack-operations tasks_from: enable_ssh

  • import_role:

name: ansible-role-openstack-operations tasks_from: restore_galera

  • import_role:

name: ansible-role-openstack-operations tasks_from: disable_ssh

  • import_role:

name: ansible-role-openstack-operations tasks_from: validate_galera 22

Restoring the overcloud

slide-23
SLIDE 23

INSERT DESIGNATOR, IF NEEDED 23

Overcloud restore demo

slide-24
SLIDE 24

INSERT DESIGNATOR, IF NEEDED 24

Trilio User workloads Freezer

slide-25
SLIDE 25

INSERT DESIGNATOR, IF NEEDED 25

  • Testing.
  • Adapting the tasks to several versions and services.
  • Maintenance over new releases.

Challenges

slide-26
SLIDE 26

INSERT DESIGNATOR, IF NEEDED 26

  • Including the ansible tasks per service configuration template.
  • Create an additional repository to store the backup/restore

workflow.

  • Composable backups.
  • Each squad testing their own backup/restore methodology.
  • Create a new CLI option to backup the Overcloud controllers?

○

  • penstack overcloud backup --controllers
  • TripleO UI options?

Ideas

slide-27
SLIDE 27

THANK YOU

plus.google.com/+RedHat linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHatNews