Nobody ever regretted making a backup AsiaBSDCon 2013 a tutorial, - - PDF document

Nobody ever regretted making a backup

AsiaBSDCon 2013 a tutorial, by Dan Langille – http://langille.org/

Tutorial Description:

Nobody ever got famous writing backup software, yet backups form one of the fundamental basics of system administration. There are many solutions to this simple problem:

• do nothing
• ad hoc solutions
• open source packages
• commercial packages

Not only is it important to backup data securely and reliably, you need to be able to restore the data. You need to know what was backed up, from what system, when, and where is that backup now. In the past decade, Bacula has been gaining ground on more established solutions, both open source and proprietary. This talk will introduce you to Bacula, show you the main components, give you an

• utline of how it works, and illustrate why Bacula is becoming so widespread.

In this tutorial, you will learn the basics of Bacula, a leading open source backup solution. As a Bacula developer, Dan has some unique insights into the use and deployment of Bacula. An avid user since 2004, he has used Bacula for his own networks and in commercial settings. Topics to cover will include:

• overview of Bacula: client, storage, director
• Jobs
• Schedules
• Catalogs
• Pools
• FileSets
• console commands
• the various retention settings and how they affect your catalog
• which database is best for use with Bacula
• the difference between a Bacula volume and a disk volume
• running a job
• restoring a job
• automatic pruning
• recycling of Volumes
• using tape libraries
• copy/migrate jobs from one media to another
• spooling

• verview of Bacula: client, storage, director

Key point: The Director (DIR) contacts the File Daemon (FD/client) and says: backup this FileSet to this Storage Daemon (SD). Vital: allow TCP connections between those components (i.e. hosts.allow, firewall, etc) Heartbeat: keep alive signal for long running jobs under some operating systems / firewalls. Can be specified on both SD & FD. A single DIR can run many hundreds/thousands of clients. Your limiting factor is SD. You can have many SD, easily. You can run many Catalog too.

Command line tools:

• bconsole
• bcopy
• bextract
• bregex
• bscan
• bsmtp
• btape
• btraceback
• bwild
• dbcheck

Context

• Unless otherwise specified, assume all resources mentioned are entries in bacula-dir.conf

SD and FD configuration

• Every SD and FD needs at least one entry like this:

Director { Name = your-dir Password = "Jg6JfArzRtVUyNNELcXF6tdrTpRvi" }

• The above line identifies the DIR which is allowed to contact this FD or SD and the name and

password it must use.

• NOTE: the above is the DIR name, not the FD/SD name. But it is the password for the FD/SD

as found in the bacula-dir.conf file.

• Thus, every password is stored in two locations:
• In the FD/SD configuration file
• In the bacula-dir.conf file
• This is a “shared secret” between the FD/SD and the DIR
• The DIR will have a similar entry within a Client (for the FD) or Storage (for the SD) resource

within bacula-dir.conf

Storage { Name = MySD Address = storage1.example.org SDPort = 9103 Password = "same password as in bacula-sd.conf" Device = FileStorage Media Type = File } Client { Name = laptop-fd Address = laptop.example.org Catalog = MyCatalog Password = "same password as in bacula-fd.conf" File Retention = 3 years Job Retention = 3 years }

• Knowing the above will save you much grief when adding new FD/SD.

Client

• Entry in bacula-dir.conf file
• identifies a client to backup
• A Client runs bacula-fd.conf
• Client resource identifies the Address of the Client (i.e. hostname)
• Catalog to use for that Client
• Password that this DIR must use to contact that client
• Other DIR might have different password
• Retention periods (File & Job; more on retention later)

Client { Name = nyi-fd Address = nyi.example.org FDPort = 9102 Catalog = MyCatalog Password = "ICDf+XmbWH3Mw753xbc6hv3gQbNc6WjyAP" TLS Require = yes TLS Enable = yes TLS CA Certificate File = /home/bacula/certificates/ca.crt TLS Certificate = /home/bacula/certificates/client.crt TLS Key = /home/bacula/certificates/client.key File Retention = 3 years Job Retention = 3 years }

Jobs

• A job runs on exactly one client
• A job consists of exactly one FileSet
• A job backs up to exactly one SD
• A job has just one schedule, which can be simple or complex
• You can have multiple jobs per client

Job { Name = "nyi basic" JobDefs = "DefaultJobRemote" Client = "nyi-fd" FileSet = "basic backup" Write Bootstrap = "/home/bacula/working/nyi-fd-basic.bsr" } Job { Name = "nyi" JobDefs = "DefaultJobRemote" Client = "nyi-fd" FileSet = "Production Web Server" Write Bootstrap = "/home/bacula/working/nyi-fd-web.bsr" }

JobDefs

• Many jobs are similar
• group the similarities into a JobDefs (i.e. Job Defaults)
• Common job definitions include
• Type – usually backup
• Level (full, incremental, differential)
• Schedule (when to run Jobs using this JobDefs)
• Storage (where to put the backups)
• Pool (what Pool to use)
• FileSet (what to backup)
• Use a JobDef so your Job really specifies how this Job differs from other Jobs, for example, a

minimal Job could be:

• Name
• JobDefs
• Client

JobDefs { Name = "DefaultJobRemote" Type = Backup Level = Incremental Client = ngaio-fd FileSet = "Full Set" Schedule = "WeeklyCycle" Storage = MegaFile Messages = Standard Pool = FullFile # required parameter for all Jobs Full Backup Pool = FullFile Differential Backup Pool = DiffFile Incremental Backup Pool = IncrFile Priority = 20 Spool Data = no Spool Attributes = yes }

Job Level

• Full – backup everything according to the FileSet
• Incremental - all files specified in the FileSet that have changed since the last successful backup
• f the the same Job using the same FileSet and Client
• Differential - all files specified in the FileSet that have changed since the last successful Full

backup of the same Job and client will be backed up

• From the Bacula FAQ:
• Before doing an Incremental or a Differential backup, Bacula checks to see if there was

a prior Full backup of the same Job that terminated successfully. If so, it uses the date that full backup started as the time for comparing if files have changed. If Bacula does not find a successful full backup, it proceeds to do one.

• Don't let your Incremental expiry before your Differential or Full
• Don't let your Differential expire before your Full
• More on retention periods later.

Schedules

• Jobs are run automatically according to the schedule assigned to that job
• A Schedules can be used by zero or more jobs
• A Schedule can indicate that a job is never run automatically (i.e. manually only)

Schedule { Name = "WeeklyCycle" Run = Level=Full 1st sun at 5:55 Run = Level=Differential 2nd-5th sun at 5:55 Run = Level=Incremental mon-sat at 5:55 } Schedule { Name = "Never" }

FileSet

• A FileSet is a list of files to backup
• A FileSet can be used by zero or more jobs
• Exactly one FileSet per job
• can specify files to exclude
• by default, does not span mount points (can be overridden but use with care; e.g. NFS)
• Be sure to enclose path names in “quotes” for Windows clients
• test your FileSet using the estimate command in bconsole

FileSet { Name = "basic backup" Include { Options { signature=MD5 } Exclude Dir Containing = .NOBACKUP File = /boot File = /usr/src/sys/i386/conf File = /etc File = /usr/local/etc File = /usr/local/info File = /usr/local/libexec/nagios/ File = /usr/local/var File = /root File = /var/db/ports File = /var/log File = /var/cron } } FileSet { Name = "ngaio files" Include { Options { signature=MD5 verify=pnugsmcs5 } File = /usr/websites File = /home/dan File = /home/bacula File = /usr/jails/flavours File = /usr/local/libexec/nagios } Exclude { File = *~ File = /home/dan/tmp File = /home/dan/ports File = /home/bacula/spooling File = /home/bacula/volumes File = /home/bacula/bacula-restores } }

Volume

• A Volume is a place to put a backup
• It may be disk, tape (DVD – not really supported any more)
• Bacula treats disk and tape the same (more or less)
• A backup resides in exactly one Volume
• May be automatically labeled
• Makes most sense for disk Volumes and tape Volumes
• To implement: LabelMedia = yes in Device resource in bacula-sd.conf
• For disk, you need LabelFormat in Pool resource

Pool

• A Pool is a collection of Volumes with similar attributes
• A Volume is created based upon a Pool definition
• You can have multiple Pools
• A Volume must belong to exactly one Pool
• The common Pool attributes are:
• Name
• Pool Type (usually Backup)
• Recycle (yes/no)
• Volume Retention
• Storage (what SD is this Pool located at?)
• LabelFormat (not recommended for bar code enabled tape libraries)
• The Pool definition is the template for a new Volume. It defines the attributes of each new

Volume from that Pool

• If you edit the Pool resource in bacula-dir.conf, the 'update' command in bconsole will update

the Pool definition. This does not affect existing Volumes. The 'update' command can also update the Volumes from the Pool definition.

Pool { Name = FullFile Pool Type = Backup Recycle = yes AutoPrune = yes Volume Retention = 3 years Storage = MegaFile Next Pool = Fulls Maximum Volume Bytes = 5G LabelFormat = "FullAuto-" } Pool { Name = DiffFile Pool Type = Backup Recycle = yes AutoPrune = yes Volume Retention = 6 weeks Storage = MegaFile Next Pool = Differentials Maximum Volume Bytes = 5G LabelFormat = "DiffAuto-" } Pool { Name = IncrFile Pool Type = Backup Recycle = yes AutoPrune = yes Volume Retention = 3 weeks Storage = MegaFile Next Pool = Incrementals Maximum Volume Bytes = 5G LabelFormat = "IncrAuto-" }

Catalogs

• The Catalog is a list of what was backed up, when, and from what client
• The Catalog is stored in a Database

Catalog { Name = MyCatalog dbname = bacula; dbaddress = localhost; user = bacula; password = "" # MultipleConnections = yes # DB Address = ngaio.example.org # DB Port = 5432 }

• Data within the Catalog includes:
• what Jobs were run
• the FileSet used
• the list of files that were backed up
• ptional checksum of each file
• where that backup is located
• what client it was run on
• list of Pools
• list of Volumes in that Pool
• Retention determines how long entries are retained in the Catalog
• Retention is only indirectly related to how long backups will remain within a Volume
• Backups might still be available after Retention expires, but don't count on it
• bextract is your best tool for backup retrieval after Retention expires; I have never used it and I

wish I never have to.

• Catalogs grow. They can grow to be large. Disk space is cheap. Time, especially when

restoring, is not cheap. Thus, I let my Catalog grow knowing it will save me time when I need to restore.

• Data is manually removed from the Catalog via the prune and purge processes.
• Pruning – removes data from the Catalog based upon Retention times
• Purging – removes data from the Catalog, completely ignoring Retention times (e.g. rm
• rf)

Catalog – backups

• Your Catalog is your best tool
• Heavily used for restores
• Without your Catalog, what you have it about the same as a tarball, more or less.
• The Catalog knows where everything is and constructs the right procedure to restore it properly.
• Backup your Catalog. To multiple locations.
• Catalogs can be reconstructed by using bscan to read all your Volumes, but this is a lengthy and

troublesome process

• I try to avoid this situation by copying a dump of the database, and the *.conf files to multiple

locations (different servers, different cities) to use in case the bacula server dies a horrible death.

# Backup the catalog database (after the nightly save) Job { Name = "BackupCatalog" JobDefs = "DefaultJob" Client = ngaio-fd # runs on db server, but not necessarily Level = Full FileSet = "Catalog" Schedule = "DailyAfterBackup" # This creates an ASCII copy of the catalog # but you can use any process you want RunBeforeJob = "/home/bacula/make_catalog_backup.new MyCatalog" # let's not delete the dump.... # This deletes the copy of the catalog # RunAfterJob = "/home/bacula/delete_catalog_backup" Write Bootstrap = "/home/bacula/working/BackupCatalog.bsr" # no sense spooling local data Spool Data = no Spool Attributes = no Priority = 100 # run after main backup } # This schedule does the catalog. Schedule { Name = "DailyAfterBackup" Run = Level=Full at 8:15 }

recycling of Volumes

• Bacula will do everything it can to not overwrite a Volume
• It will create new Volumes (ask for new tapes), rather than overwrite a Volume
• Overwriting is known as Recycling
• Learn the Bacula Recycling Algorithm (it is in the documentation) (following is taken from it):
• Automatic recycling of Volumes is controlled by four records in the Pool resource

definition in the Director's configuration file. These four records are:

• AutoPrune = yes
• VolumeRetention = time
• Recycle = yes
• RecyclePool = SomeOtherPoolPerhapsScratch
• The above three directives are all you need, assuming that you fill each of your Volumes,

then wait the Volume Retention period before reusing them. If you want Bacula to stop using a Volume and recycle it before it is full, you will need to use one or more additional directives such as:

• Use Volume Once = yes
• Volume Use Duration = ttt
• Maximum Volume Jobs = nnn
• Maximum Volume Bytes = mmm
• Automatic recycling of Volumes is performed by Bacula only when it wants a new Volume and

no appendable Volumes are available in the Pool in questoin. Bacula will then search the Pool for any Volumes with the Recycle flag set and with Volume Status = Purged. At that point, Bacula will choose the oldest purged volume, and recycle it.

• No Purged Volumes? Bacula will search the Catalog for Volumes which have been pruned of all
• Jobs. Only Volumes marked Full or Used will be searched. The Volume then gets purged if the

Volume Retention period has expired. If found, go back to the previous bullet point and continue. Data on Volume remains intact until Recycling occurs. That is, the Volume may be pruned and/or purged, and the backup is still there, you can just can't easily get the data off with a standard restore command (see bextract if you need that data)

the various retention settings and how they affect your catalog

• Volume Retention – defined in the Pool Resource
• File Retention
• defined in Client resource
• defines the length of time that Bacula will keep File records in the Catalog after the Jo

End time

• default is 60 days
• File records may be retained for a shorter period if either a Job or Volume retention is
• shorter. The shortest retention period of the three takes precedence.
• Job Retention
• defined in Client resource
• defines the length of time that Bacula will keep Job records in the Catalog after the Job

End time

• default is 180 days
• If a Job is selected for pruning, all associated File records will also be pruned.
• Thus, File Retention > Job Retention achieves nothing.
• Jobs may be retained for a shorter period if Volume Retention is shorter.
• Retention affects the Catalog data, not backups on the Volume <= very important!
• Once the Catalog does not contain data for a Volume, that Volume can be recycled, but this does

not mean that the Volume WILL be recycled. See Bacula Recycling Algorithm

• If File Retention has passed, you cannot restore individual files from a backup. You must

restore the whole backup.

• If Job retention has passed, you cannot restore a backup (using the restore command in

bconsole).

• If Job Retention has passed, and you need to restore that backup, you need to use bextract (ugly

command line tool, which is not easy to use; restore in bconsole IS easy)

• The settings I use are:
• Job Retention = 3 Years
• File Retention = 3 Years
• Volume Retention = variable depending on goal of Pool
• I suggest always having File = Job Retention

bacula-fd.conf configuration

• A given client installation (bacula-fd) can be used by multiple bacula-dir
• That is: a client, say Client1, can take instructions from zero or more Directors
• Each director is listed separately:

Director { Name = foo Password = "FooPassword" } Director { Name = bar Password = "BarPassword" }

• This client will accept instructions from either of the above two Directors, each of which must

supply the indicated password

• Each Director will have a Client resource in their respective bacula-dir.conf file which looks

like:

# from foo's bacula-dir.conf Client { Name = Client1 Password = "FooPassword" Address = client1.example.org } # from bar's bacula-dir.conf Client { Name = Client1 Password = "BarPassword" Address = client1.example.org }

• Bacula passwords are plain test. Not encrypted. They only look like they are. Relies upon the

OS for security of .conf files.

• When you get told that that the passwords / names do not match (and you will get that error),

check both the name and the password in the two configuration files match.

bacula-sd.conf configuration

• Much like the Client resource, each Storage resource in bacula-dir.conf contains an Address and

a Password. That password will also appear in the bacula-sd.conf file for the bacula-sd running at that Address.

• A given bacula-sd can handle multiple Storage resources; that is, several Storage resources can

refer to the same Address value. A given SD can backup to tape or disk.

• An SD which is mutli-homed, can have two different Storage resources in bacula-dir.conf, each

with a different Address, but each refers to the same bacula-sd, listening on multiple addresses. This is useful for internal and external Clients, although I prefer to use a VPN for external

• Clients. OpenVPN is my VPN of choice.
• This is in the bacula-dir.conf file:

Storage { Name = MegaFile # this name known only within bacula-dir Address = kraken.example.org SDPort = 9103 Password = "SuperSecretPassword" Device = MegaFile # this name appears in bacula-sd.conf Media Type = File Maximum Concurrent Jobs = 40 }

• This is in the bacula-sd.conf file:

Storage { Name = kraken-sd SDPort = 9103 SDAddress = 10.0.0.12 # this host has multiple IP, listen here WorkingDirectory = "/storage/compressed/bacula/working" Pid Directory = "/var/run" Maximum Concurrent Jobs = 20 } Director { Name = bacula-dir Password = "SuperSecretPassword" } Device { Name = MegaFile Media Type = File Archive Device = /storage/compressed/bacula/volumes LabelMedia = yes Random Access = yes AutomaticMount = yes RemovableMedia = no AlwaysOpen = no }

console commands

• I use, and recommend, bconsole
• It works. Highly tested. Most reliable.
• . <= escape charater; use it to get out of a command
• status – what's happening on a client, storage, or director
• run
• restore
• m
• help

which database is best for use with Bacula

• As the author of the PostgreSQL backend, I always prefer PostgreSQL
• license is simple
• I think it is better suited to bigger databases and higher loads
• Use MySQL if you cannot use PostgreSQL
• I do not recommend SQLite for use with Bacula, other than some light testing and it is entirely

unsuitable for production

$ psql bacula psql (8.4.12, server 8.4.13) Type "help" for help. bacula=# \dt List of relations Schema | Name | Type | Owner

• -------+----------------+-------+--------

public | basefiles | table | bacula public | cdimages | table | bacula public | client | table | bacula public | counters | table | bacula public | device | table | bacula public | file | table | bacula public | filename | table | bacula public | fileset | table | bacula public | job | table | bacula public | jobhisto | table | bacula public | jobmedia | table | bacula public | location | table | bacula public | locationlog | table | bacula public | log | table | bacula public | media | table | bacula public | mediatype | table | bacula public | path | table | bacula public | pathhierarchy | table | bacula public | pathvisibility | table | bacula public | pool | table | bacula public | restoreobject | table | bacula public | status | table | bacula public | storage | table | bacula public | unsavedfiles | table | bacula public | version | table | bacula (25 rows) bacula=#

Database scripts

• Scripts for creating / maintaining the database are included with Bacula:

[dan@ngaio:/usr/local/share/bacula] $ grep -l psql * create_postgresql_database drop_postgresql_tables grant_postgresql_privileges make_catalog_backup make_postgresql_tables update_postgresql_tables

• But don't use those scripts, use the parent scripts:

$ ls *bacula* create_bacula_database drop_bacula_tables grant_bacula_privileges make_bacula_tables drop_bacula_database update_bacula_tables

• Be sure to run those commands as the bacula user, or you might wind up with tables not owned

by bacula. Mind you, I run them as me, then tidy up the db later...

the difference between a Bacula volume and a disk volume

• Not much
• Bacula treats them the same, more or less
• For file Volumes, Bacula creates a file with the same name as the label
• Newbies run into disk space problems because they haven't monitored the free disk space and

fail to implement a strategy.

• See Limiting disk space (later in this document)

running a job

$ bconsole Connecting to Director bacula.unixathome.org:9101 1000 OK: bacula-dir Version: 5.2.12 (12 September 2012) Enter a period to cancel a command. *run job=dent Run Backup job JobName: dent Level: Incremental Client: dent-fd FileSet: dent files Pool: FullFile (From Job resource) Storage: MegaFile (From Pool resource) When: 2013-01-27 17:41:32 Priority: 10 OK to run? (yes/mod/no): yes Job queued. JobId=118611 *

*m 27-Jan 17:41 bacula-dir JobId 118611: Start Backup JobId 118611, Job=dent.2013-01- 27_17.41.34_45 27-Jan 17:41 bacula-dir JobId 118611: Using Device "MegaFile" 27-Jan 17:41 kraken-sd JobId 118611: Volume "Incr-2341" previously written, moving to end of data. 27-Jan 17:41 kraken-sd JobId 118611: Ready to append to end of Volume "IncrAuto- 2341" size=2922249218 27-Jan 17:41 kraken-sd JobId 118611: Elapsed time=00:00:01, Transfer rate=95.47 K Bytes/second 27-Jan 17:41 kraken-sd JobId 118611: Sending spooled attrs to the Director. Despooling 999 bytes ... 27-Jan 17:41 bacula-dir JobId 118611: Bacula bacula-dir 5.2.12 (12Sep12): Build OS: i386-portbld-freebsd8.2 freebsd 8.2-STABLE JobId: 118611 Job: dent.2013-01-27_17.41.34_45 Backup Level: Incremental, since=2013-01-26 23:30:03 Client: "dent-fd" 5.0.3 (04Aug10) i386-apple- darwin10.7.3,osx,10.7.3 FileSet: "dent files" 2011-04-03 02:05:58 Pool: "IncrFile" (From Job IncPool override) Catalog: "MyCatalog" (From Client resource) Storage: "MegaFile" (From Pool resource) Scheduled time: 27-Jan-2013 17:41:32 Start time: 27-Jan-2013 17:41:39 End time: 27-Jan-2013 17:41:43 Elapsed time: 4 secs Priority: 10 FD Files Written: 4 SD Files Written: 4 FD Bytes Written: 95,017 (95.01 KB) SD Bytes Written: 95,470 (95.47 KB) Rate: 23.8 KB/s Software Compression: None VSS: no Encryption: no Accurate: no Volume name(s): Incr-2341 Volume Session Id: 248 Volume Session Time: 1358896184 Last Volume Bytes: 2,922,345,228 (2.922 GB) Non-fatal FD errors: 0 SD Errors: 0 FD termination status: OK SD termination status: OK Termination: Backup OK 27-Jan 17:41 bacula-dir JobId 118611: Begin pruning Jobs older than 3 years . 27-Jan 17:41 bacula-dir JobId 118611: No Jobs found to prune. 27-Jan 17:41 bacula-dir JobId 118611: Begin pruning Files. 27-Jan 17:41 bacula-dir JobId 118611: No Files found to prune. 27-Jan 17:41 bacula-dir JobId 118611: End auto prune. *

• can modify job parameters

*run job=dent Run Backup job JobName: dent Level: Incremental Client: dent-fd FileSet: dent files Pool: FullFile (From Job resource) Storage: MegaFile (From Pool resource) When: 2013-01-27 18:54:18 Priority: 10 OK to run? (yes/mod/no): mod Parameters to modify: 1: Level 2: Storage 3: Job 4: FileSet 5: Client 6: When 7: Priority 8: Pool 9: Plugin Options Select parameter to modify (1-9): 1 Levels: 1: Full 2: Incremental 3: Differential 4: Since 5: VirtualFull Select level (1-5): 3 Run Backup job JobName: dent Level: Differential Client: dent-fd FileSet: dent files Pool: FullFile (From Job resource) Storage: MegaFile (From Pool resource) When: 2013-01-27 18:54:18 Priority: 10 OK to run? (yes/mod/no): yes Job queued. JobId=118613 *

restoring a job

• You need just one restore Job
• You can override all Job attributes at run time
• Lots of restore options
• Mark files you want
• Restore to a different client
• Restore to a different location on that client (e.g. not /, but /tmp, for example)

*restore client=dent-fd First you select one or more JobIds that contain files to be restored. You will be presented several methods

• f specifying the JobIds. Then you will be allowed to

select which files from those JobIds are to be restored. To select the JobIds, you have the following choices: 1: List last 20 Jobs run 2: List Jobs where a given File is saved 3: Enter list of comma separated JobIds to select 4: Enter SQL list command 5: Select the most recent backup for a client 6: Select backup for a client before a specified time 7: Enter a list of files to restore 8: Enter a list of files to restore before a specified time 9: Find the JobIds of the most recent backup for a client 10: Find the JobIds for a backup for a client before a specified time 11: Enter a list of directories to restore for found JobIds 12: Select full restore to a specified Job date 13: Cancel Select item: (1-13): 5 Automatically selected FileSet: dent files +---------+-------+----------+-------------+---------------------+---------------+ | jobid | level | jobfiles | jobbytes | starttime | volumename | +---------+-------+----------+-------------+---------------------+---------------+ | 113,257 | F | 1,008 | 855,874,316 | 2012-12-03 22:38:28 | FullAuto-3415 | | 113,257 | F | 1,008 | 855,874,316 | 2012-12-03 22:38:28 | FullAuto-3414 | | 115,149 | D | 75 | 1,051,488 | 2012-12-23 23:53:17 | DiffAuto-2389 | | 115,377 | I | 5 | 11,210 | 2012-12-25 23:30:06 | IncrAuto-2350 | | 115,475 | I | 2 | 33,346 | 2012-12-26 23:30:04 | IncrAuto-2350 | | 115,567 | I | 4 | 29,653 | 2012-12-28 00:53:58 | IncrAuto-2351 | | 115,822 | I | 6 | 48,977 | 2012-12-31 00:27:34 | IncrAuto-1405 | | 115,925 | I | 7 | 46,367 | 2012-12-31 23:30:04 | IncrAuto-1400 | | 116,011 | I | 2 | 46,027 | 2013-01-01 23:30:10 | IncrAuto-1400 | | 116,360 | I | 11 | 535,729 | 2013-01-05 23:30:08 | IncrAuto-1401 | | 117,312 | I | 6 | 120,189 | 2013-01-14 00:27:47 | IncrAuto-1411 | | 117,547 | I | 6 | 734,831 | 2013-01-15 23:30:04 | IncrAuto-1413 | | 117,589 | I | 3 | 718,868,869 | 2013-01-16 23:30:04 | IncrAuto-2353 | | 117,631 | I | 61 | 3,066,032 | 2013-01-17 23:30:04 | IncrAuto-2353 | | 118,241 | I | 6 | 47,015 | 2013-01-21 23:30:14 | IncrAuto-2360 | | 118,341 | I | 1 | 6,669 | 2013-01-22 23:30:05 | IncrAuto-2360 | | 118,568 | I | 9 | 162,064 | 2013-01-26 21:37:19 | IncrAuto-2341 | | 118,570 | I | 1 | 7,103 | 2013-01-26 23:30:03 | IncrAuto-2341 | | 118,611 | I | 4 | 95,017 | 2013-01-27 17:41:39 | IncrAuto-2341 | +---------+-------+----------+-------------+---------------------+---------------+ You have selected the following JobIds: 113257,115149,115377,115475,115567,115822,115925,116011,116360,117312,117547,117589 ,117631,118241,118341,118568,118570,118611 Building directory tree for JobId(s) 113257,115149,115377,115475,115567,115822,115925,116011,116360,117312,117547,117589 ,117631,118241,118341,118568,118570,118611 ... +++++++++++++++++++++++++++++++++++ +++++++++++ 985 files inserted into the tree. You are now entering file selection mode where you add (mark) and remove (unmark) files to be restored. No files are initially added, unless you used the "all" keyword on the command line.

Enter "done" to leave this mode. cwd is: / $ ls Users/ $ cd Users cwd is: /Users/ $ ls dan/ $ cd dan cwd is: /Users/dan/ $ mark *.jpg 8 files marked. $ done Bootstrap records written to /home/bacula/working/bacula-dir.restore.27.bsr The job will require the following Volume(s) Storage(s) SD Device(s) =========================================================================== FullAuto-3414 MegaFile MegaFile FullAuto-3415 MegaFile MegaFile IncrAuto-2360 MegaFile MegaFile Volumes marked with "*" are online. 7 files selected to be restored. Run Restore job JobName: RestoreFiles Bootstrap: /home/bacula/working/bacula-dir.restore.27.bsr Where: /tmp/bacula-restores Replace: always FileSet: Full Set Backup Client: dent-fd Restore Client: dent-fd Storage: MegaFile When: 2013-01-27 19:02:24 Catalog: MyCatalog Priority: 10 Plugin Options: *None* OK to run? (yes/mod/no): yes Job queued. JobId=118614 *

*m 27-Jan 19:11 bacula-dir JobId 118614: Start Restore Job RestoreFiles.2013-01- 27_19.02.28_49 27-Jan 19:11 bacula-dir JobId 118614: Using Device "MegaFile" 27-Jan 19:11 kraken-sd JobId 118614: Ready to read from volume "FullAuto-3414" on device "MegaFile" (/storage/compressed/bacula/volumes). 27-Jan 19:11 kraken-sd JobId 118614: Forward spacing Volume "FullAuto-3414" to file:block 1:328378838. 27-Jan 19:11 kraken-sd JobId 118614: End of Volume at file 1 on device "MegaFile" (/storage/compressed/bacula/volumes), Volume "FullAuto-3414" 27-Jan 19:11 kraken-sd JobId 118614: Ready to read from volume "FullAuto-3415" on device "MegaFile" (/storage/compressed/bacula/volumes). 27-Jan 19:11 kraken-sd JobId 118614: Forward spacing Volume "FullAuto-3415" to file:block 0:216. 27-Jan 19:11 kraken-sd JobId 118614: End of Volume at file 0 on device "MegaFile" (/storage/compressed/bacula/volumes), Volume "FullAuto-3415" 27-Jan 19:11 kraken-sd JobId 118614: Ready to read from volume "IncrAuto-2360" on device "MegaFile" (/storage/compressed/bacula/volumes). 27-Jan 19:11 kraken-sd JobId 118614: Forward spacing Volume "IncrAuto-2360" to file:block 0:726468763. 27-Jan 19:11 bacula-dir JobId 118614: Bacula bacula-dir 5.2.12 (12Sep12): Build OS: i386-portbld-freebsd8.2 freebsd 8.2-STABLE JobId: 118614 Job: RestoreFiles.2013-01-27_19.02.28_49 Restore Client: dent-fd Start time: 27-Jan-2013 19:11:45 End time: 27-Jan-2013 19:11:52 Files Expected: 7 Files Restored: 8 Bytes Restored: 3,049,421 Rate: 435.6 KB/s FD Errors: 0 FD termination status: OK SD termination status: OK Termination: Restore OK 27-Jan 19:11 bacula-dir JobId 118614: Begin pruning Jobs older than 3 years . 27-Jan 19:11 bacula-dir JobId 118614: No Jobs found to prune. 27-Jan 19:11 bacula-dir JobId 118614: Begin pruning Files. 27-Jan 19:11 bacula-dir JobId 118614: No Files found to prune. 27-Jan 19:11 bacula-dir JobId 118614: End auto prune. [dan@dent:/tmp/bacula-restores] $ sudo ls Users/dan 1.jpg 3.jpg IMG_2760.jpg load-bars.jpg 2.jpg 4.jpg devsummit127TM.jpg tires.jpg [dan@dent:/tmp/bacula-restores] $

automatic pruning

• The Catalog keeps growing. Pruning removes outdata entries according to Retention periods.
• To automatically prune, recommended as part of your Pool:
• AutoPrune = yes

27-Jan 17:41 bacula-dir JobId 118611: Begin pruning Jobs older than 3 years . 27-Jan 17:41 bacula-dir JobId 118611: No Jobs found to prune. 27-Jan 17:41 bacula-dir JobId 118611: Begin pruning Files. 27-Jan 17:41 bacula-dir JobId 118611: No Files found to prune. 27-Jan 17:41 bacula-dir JobId 118611: End auto prune.

manual pruning

*prune volume=ETU228 Automatically selected Catalog: MyCatalog Using Catalog "MyCatalog" The current Volume retention period is: 28 days Continue? (yes/mod/no): yes There are no more Jobs associated with Volume "ETU228". Marking it purged. New Pool is: Scratch * *prune volume=ETU135 The current Volume retention period is: 3 years Continue? (yes/mod/no): yes *

using tape libraries

• Tape libraries, like tape drives, are accessed via SCSI interface
• If your OS can talk to the tape library, then Bacula can
• look at mtx-changer script, and customize it
• mtx-changer is the go-between for Bacula and your OS
• ensure bacula user can run all the commands required
• alter permissions on devices if required
• r add bacula to the appropriate groups if appropriate
• run btape tests
• run a backup which spans tapes, then restore that backup and compare via diff
• Take your time setting up a new tape library unit. Get it right.
• Might take a day or two, because testing tapes might take a while, not to mention permissions,

can be persnickety.

• My experiences with tape libraries:
• http://www.freebsddiary.org/tape-library-integration.php
• http://www.freebsddiary.org/tape-library.php

• When my tapes are recycled, they go into the Scratch pool, from which other Pools can take

tapes:

Pool { Name = Scratch Pool Type = Backup } Pool { Name = Fulls Pool Type = Backup Recycle = yes AutoPrune = yes Volume Retention = 3 years Storage = DigitalTapeLibrary RecyclePool = Scratch } Pool { Name = Differentials Pool Type = Backup Recycle = yes AutoPrune = yes Volume Retention = 2 months Storage = DigitalTapeLibrary RecyclePool = Scratch } Pool { Name = Incrementals Pool Type = Backup Recycle = yes AutoPrune = yes Volume Retention = 4 weeks Storage = DigitalTapeLibrary RecyclePool = Scratch }

Limiting disk space used

• Ensuring there is enough disk space for a backup is not within the scope of Bacula
• disk space is the concern of the sysadmin
• Use nagios, etc, to report upon disk space available
• Pool resource: Maximum Volumes = N
• Pool resource: Maximum Volume Bytes = S
• Maximum space used by that Pool will be (N * S) bytes
• see http://www.bacula.org/5.0.x-manuals/en/main/main/Configuring_Director.html#SECTION0018150000000000000000
• You can also restrict Volumes with:
• UseVolumeOnce = yes
• Maximum Volume Jobs = nnn
• Volume Use Duration = ttt

copy/migrate jobs from one media to another

• A Job may copy an existing Job another Volume (Type = Copy)
• A Job may migrate an existing job (i.e. moved) to another Volume (Type = Migrate)
• Copy and Migrate can only be done within a single SD (no copying from one SD to another)
• Only Jobs in Volumes which are marked Full, Used, or Error. Volumes marked as Append will

not be considered.

• Use Copy and/or Migrate to achieve copy first to disk, then later to tape
• Main difference between Copy and Migrate is Copy does not affect the original Job, which is

still available for use.

• When reading the Migration documentation, keep in mind that although it mentions migration,

that term includes both Copy and Migrate Jobs, unless otherwise mentioned.

• Many different criteria for Copy/Migrate

• Sample copy job

Job { Name = "CopyToTape-Inc" Type = Copy Level = Incremental Pool = IncrFile FileSet = "EmptyCopyToTape" Client = kraken-fd Schedule = "CopyToTape-Inc" Storage = DigitalTapeLibrary Messages = Standard # don't spool date when backing up to tape from local disk Spool Data = no Spool Attributes = yes Maximum Concurrent Jobs = 40 Priority = 410 Selection Type = SQL Query Selection Pattern = " SELECT DISTINCT J.JobId, J.StartTime FROM Job J, Pool P WHERE P.Name = 'IncrFile' AND P.PoolId = J.PoolId AND J.Type = 'B' AND J.JobStatus IN ('T','W') AND J.jobBytes > 0 AND J.JobId NOT IN (SELECT PriorJobId FROM Job WHERE Type IN ('B','C') AND Job.JobStatus IN ('T','W') AND PriorJobId != 0) ORDER BY J.StartTime " }

Spooling

• Spooling to local disk before writing to tape can be better than writing straight to tape
• Avoids stop/start of tape drive if network bandwidth cannot keep up with tape drive
• Do not confuse spooling with backup to disk, then copy to tape; spooling is just spooling
• Spooling data can also increase throughput; set Spool Data = yes
• If backing up to disk, always set: Spool Data = no
• You can also spool attributes; Bacula normally updates the Catalog as each file is written to the

Volume during a backup; Spooling Attributes means that the Catalog is not updated until after all the backup data has been written to the SD

• Spooling Attributes can greatly improve performance. Set 'Spool Attributes = yes' as part of Job

Resource.

• Be aware that the job will continue 'running' in bacula-dir long after the client portion of the

backup has completed. This is expected.

NOTES:

• Bacula does not use tar. For disaster recovery, use bextract, bls.
• Best practice: copy .conf and .sql files in multiple accessible locations so you never have to use

bextract.

Deduplication

• This code will be released in the Bacula Enterprise version at the end of March 2013
• For now, I recommend ZFS dedup if you want this

base jobs

• Base job designed for things that rarely change, such as 100 identical work stations, all installed

with the same OS

delta backups

• Available in Bacula Enterprise

compression

• Can be done at FD
• can be done by tape drive
• can be done by OS
• I let ZFS do my compression for disk-based Volumes

Things not to waste time on

• Labels – let Bacula label your Volumes and do not obsess over making sure that all the backups

for your laptop are in a Volume named 'laptop-2013-01-13.from.Toyko'

• Volume Names - The Catalog is the only one that needs to know the Volume name. Let it worry

about it.

• Counters - Do not obsess over counters (used to name Volumes). They may not be sequential

with a Pool so let the Catalog worry about that.