Sync, Clone, and Snapshot ... Dr Ashley Aitken Curtin University A.Aitken@Curtin.Edu.Au AshleyAitken (Twitter, Skype, LinkedIn) World 2012 XW12 1 2 Overview and Introduction XW12 XW12 3 4 Overview School of IS Curtin University • Preamble • Overview and Introduction • Teaching • Syncing with rsync • Introduction to Business IS and IT • Cloning with CCC • Object-Oriented Analysis & Design • Snapshotting with rsnapshot • Research • Agile & Lean Software Engineering • Summary • Teaching and Learning Research • Q & A XW12 XW12 5 6
Running Code Productions Disclaimers www.runningcode.com.au • These tools are known to change • Professional Training & Mentoring • Please read the manual pages • Cocoa, WebObjects, Java, OO, Unix, ... • I am not a professional SA • Software Consulting Services • Architecture, analysis, design, development ... • Intro to Intermediate Level • Early Stage Startups • YMMV • Online Payments, ... XW12 XW12 7 8 Rsync • A fast, versatile, remote (and local) file- copying tool Syncing $ rsync [options] SRC... [DEST] • Uses a delta-transform to reduce transfer ... with rsync ... times by sending only changes to files ... • Heaps of options ... see man page ... • Compresses, filters, pattern rules, batch mode, ... XW12 XW12 9 10 http://jakob.jenkov.com/ http://jakob.jenkov.com/ XW12 XW12 11 12
Other Features • support for copying links, devices, owners, groups, and permissions • exclude and exclude-from options similar to GNU tar • a CVS exclude mode for ignoring the same files that CVS would ignore • can use any transparent remote shell, including ssh or rsh • does not require super-user privileges • pipelining of file transfers to minimize latency costs http://jakob.jenkov.com/ • support for anonymous or authenticated rsync XW12 XW12 daemons (ideal for mirroring) 13 14 Remote Shell 1. Run Remote Shell or Daemon • Used when destination path contains “:” • rsync requires software running on the source and target • Two options: Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST] • 1. Run an instance of rsync in remote shell Push: rsync [OPTION...] SRC... [USER@]HOST:DEST • 2. Run rsync daemon permanently on target XW12 XW12 15 16 2. Contact Rsync Daemon Starting rsync daemon • Used when the destination path contains “::” OR when “rsync://...” URL is used $ rsync --daemon • rsync --daemon running on remote host or use launchd Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] Configuration file /etc/rsyncd.conf rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST] Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST XW12 XW12 17 18
Directory and Contents Sample Config File or Just Contents? motd file = /etc/rsyncd.motd • N O T RAILING S LASH ON SRC log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid $ rsync SRC ... lock file = /var/run/rsync.lock • Copy the specified directory (and its contents) [simple_path_name] path = /rsync_files_here • T RAILING S LASH O N SRC/ comment = My Very Own Rsync Server uid = nobody gid = nobody read only = no $ rsync SRC/ ... list = yes auth users = username • Copy the contents of the specified directory secrets file = /etc/rsyncd.scrt • SRC folder is not created at the destination XW12 XW12 19 20 Rsync on Mac OS X Solutions using Rsync • There’s history ... • Version provided .... • RsyncX • See Carbon Copy Cloner • See Rsnapshot.pl XW12 XW12 http://en.wikipedia.org/wiki/Rsync 21 22 In the transporter room ... Issues? • Stops on errors • Need to start from beginning again ... • Synchronisation is only in one direction • Unless you do it in both directions ... XW12 XW12 23 24
Carbon Copy Cloner (CCC) • Clone, Synchronize, Backup, Schedule Cloning • Uses rsync & asr (and diskutil and ....) ... with CCC ... • Full shareware (with Edu discounts) • Mac OS X Only • Mike Bombich • http://www.bombich.com/ XW12 XW12 25 26 CCC Cloning CCC Incremental Backup • Smart and fast incremental syncing • Bootable clone (if System Disk) • Hard drives, network shares, disk images • Great for migrating to new hard disk • Well thought out default settings • Supports block-level disk-to-disk clones • Uses rsync XW12 XW12 27 28 CCC Schedule CCC General • Scheduled hourly, daily, weekly, monthly • Simple, well thought-out, interface • When disk mounted (name or UUID) • Non-proprietary backups (HFS+) • Email notifications on completion • Cloning Coach • Will wake and sleep ... XW12 XW12 29 30
DEMO Underlying tech ... • Block-Level Copy • CCC uses Apple Software Restore (asr) • Incremental Sync / Copy • CCC uses included version of rsync DEMO XW12 XW12 31 32 CCC and Mac OS X Backup Bouncer Filesystem Compatibility • Rsync 3 version patched to support • Test suite for backup programs HFS+ enhancements (metadata) • Use to compare efficacy of toolls • Passes extended Backup Bouncer Test Suite https://github.com/n8gray/Backup-Bouncer http://www.bombich.com/groups/ccc/wiki/7ba51/Improvements_to_Backup_Bouncer.html http://www.n8gray.org/code/backup-bouncer/ XW12 XW12 33 34 bash-3.2# ./bbouncer verify -d /Volumes/Source/ /Volumes/Target/ ... bash-3.2# ./bbouncer verify -d /Volumes/Source/ /Volumes/Target/ ! Verifying: basic-permissions ... ok ! Sub-test: modification time ... ok ! Verifying: timestamps ... ok ! Sub-test: [ on symlinks ] ... ok ! Verifying: symlinks ... ok ! Sub-test: creation time ... ok ! Verifying: symlink-ownership ... ok ! ok ! Verifying: hardlinks ... ok ! Verifying: hfs-compression ... ! Verifying: resource-forks ... ! Sub-test: decmpfs xattr ... preserved ! Sub-test: on files ... ok ! Sub-test: UF_COMPRESSED flag ... set ! Sub-test: on hardlinked files ... ok ! Sub-test: file contents ... match ! Verifying: finder-flags ... ok ! Sub-test: creation time ... ok ! Verifying: finder-locks ... ok ! Sub-test: modification time ... ok ! Verifying: creation-date ... ok ! Sub-test: hard link inode ... ok ! Verifying: bsd-flags ... ok ! Sub-test: hard link decmpfs xattr ... preserved ! Verifying: extended-attrs ... ! Sub-test: hard link UF_COMPRESSED flag ... set ! Sub-test: [ on files ] ... ok ! Sub-test: hard link modification time ... ok ! Sub-test: creation time ... ok ! ok ! Sub-test: modification time ... ok ! Verifying: hfs-compression_large ... ! Sub-test: [ on locked files ] ... ok ! Sub-test: decmpfs xattr ... preserved ! Sub-test: creation time ... ok ! Sub-test: UF_COMPRESSED flag ... set ! Sub-test: modification time ... ok ! Sub-test: file contents ... match ! Sub-test: [ on directories ] ... ok ! Sub-test: creation time ... ok XW12 XW12 ! Sub-test: creation time ... ok ! Sub-test: modification time ... ok ! Sub-test: modification time ... ok ! Sub-test: hard link inode ... ok 35 36
... bash-3.2# ./bbouncer verify -d /Volumes/Source/ /Volumes/Target/ ! Sub-test: hard link inode ... ok ! Sub-test: hard link decmpfs xattr ... preserved Example Uses ! Sub-test: hard link UF_COMPRESSED flag ... set ! Sub-test: hard link modification time ... ok ! ok • Laptop Disk Clone ! Verifying: access-control-lists ... ! Sub-test: on files ... ok ! Sub-test: on locked files ... ok • Keeping a bootable copy of a laptop disk ! Sub-test: on dirs ... ok volume (or volumes) ! Sub-test: on non-inherited acls ... ok ! Sub-test: on inherited acls ... ok • Time Machine used as well ... ! ok ! Verifying: fifo ... ok • Mirroring Folders ! Verifying: devices ... ok ! Verifying: combo-tests ... • iTunes, iPhoto, ... ! Sub-test: xattrs + rsrc forks ... ok ! Sub-test: lots of metadata ... ok ! ok XW12 XW12 37 38 Comparison with other tools ... Snapshotting • SuperDuper ... with rsnapshot ... • ... XW12 XW12 39 40 Underlying tech ... Rsnapshot www.rsnapshot.org • Rsync uses hard links for unchanged files (creates new directories) • Automated backup based on rsync date=`date "+%Y-%m-%dT%H:%M:%S"` • Multiple snapshots of filesys trees rsync -aP --link-dest=$HOME/Backups/current /path/to/important_files $HOME/Backups/back-$date rm -f $HOME/Backups/current • Uses hard links to save space ln -s $HOME/Backups/back-$date $HOME/Backups/current • Restore using rsync • Compare with Time Machine (TM) • Perl script • HFS+ has to simulate hard links? • TM uses hard links to directories XW12 XW12 41 42
Configuration File Scheduling • Launchd • Tab delimited (not spaces) • Crontab (deprecated) • Filesys trees are specified in rsync module format • ? XW12 XW12 43 44 Rsnapshot Commands Hourly Example • Daily • /usr/bin/rsnapshot -c /etc/rsnapshot/xyz.rsnapshot.conf daily 2>&1 | /usr/local/bin/rsnapreport.pl | mail -s "XYZ Daily Backup" admin • Weekly • /usr/bin/rsnapshot -c /etc/rsnapshot/xyz.rsnapshot.conf weekly 2>&1 | /usr/local/bin/rsnapreport.pl | mail -s "XYZ Weekly Backup" admin XW12 XW12 45 46 Daily Example Disk Usage XW12 XW12 47 48
Recommend
More recommend
