GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → adlibre → adlibre-backup

adlibre / adlibre-backup

Licence: BSD-3-Clause license
High performance rsync backup utilising BTRFS / ZFS filesystem features

Programming Languages

shell
77523 projects

Labels

backuprsyncsnapshotbtrfszfsbackup-serverdedup

Projects that are alternatives of or similar to adlibre-backup

btrfs-backup
A simple, flexible script for versioned backups using btrfs and rsync
Stars: ✭ 59 (+68.57%)
Mutual labels:  backup, rsync, snapshot, btrfs
BaNG
Backup Next Generation for Linux & Mac using rsync (support hardlinks and btrfs snapshots), Web-Frontend, Statistics, History-Merger)
Stars: ✭ 28 (-20%)
Mutual labels:  backup, rsync, snapshot, btrfs
Backintime
Back In Time - A simple backup tool for Linux
Stars: ✭ 1,066 (+2945.71%)
Mutual labels:  backup, rsync, snapshot
Btrbk
Tool for creating snapshots and remote backups of btrfs subvolumes
Stars: ✭ 605 (+1628.57%)
Mutual labels:  backup, snapshot, btrfs
Btrfs Sxbackup
Incremental btrfs snapshot backups with push/pull support via SSH
Stars: ✭ 105 (+200%)
Mutual labels:  backup, snapshot, btrfs
planb
PlanB - automating remote backups and snapshots with zfs/rsync
Stars: ✭ 24 (-31.43%)
Mutual labels:  backup, rsync, zfs
aws-tag-sched-ops
Retired, please see https://github.com/sqlxpert/lights-off-aws
Stars: ✭ 24 (-31.43%)
Mutual labels:  backup, snapshot
Backy2
backy2: Deduplicating block based backup software for ceph/rbd, image files and devices
Stars: ✭ 126 (+260%)
Mutual labels:  backup, snapshot
Rsync Incremental Backup
Configurable bash script to send incremental backups of your data to a local or remote target
Stars: ✭ 150 (+328.57%)
Mutual labels:  backup, rsync
Rsnapshot
a tool for backing up your data using rsync (if you want to get help, use https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss)
Stars: ✭ 2,335 (+6571.43%)
Mutual labels:  backup, rsync
Rsyncosx
A macOS GUI for rsync
Stars: ✭ 780 (+2128.57%)
Mutual labels:  backup, rsync
Knoxite
A data storage & backup system
Stars: ✭ 165 (+371.43%)
Mutual labels:  backup, snapshot
Rsync Time Backup
Time Machine style backup with rsync.
Stars: ✭ 2,572 (+7248.57%)
Mutual labels:  backup, rsync
Cv4pve Barc
Backup And Restore Ceph for Proxmox VE
Stars: ✭ 74 (+111.43%)
Mutual labels:  backup, snapshot
Hactar
📃 An incremential daily backup script using rsync
Stars: ✭ 34 (-2.86%)
Mutual labels:  backup, rsync
System Tar And Restore
Backup and Restore your system using tar or Transfer it with rsync
Stars: ✭ 170 (+385.71%)
Mutual labels:  backup, rsync
virt-backup
Fully backup your KVM Virtual Machines
Stars: ✭ 27 (-22.86%)
Mutual labels:  backup, snapshot
Elkarbackup
Open source backup solution for your network
Stars: ✭ 247 (+605.71%)
Mutual labels:  backup, rsync
sicksync
Don’t accept the available as the preferable. Go the extra mile with extra speed.
Stars: ✭ 67 (+91.43%)
Mutual labels:  backup, rsync
Rdiff Backup
Reverse differential backup tool, over a network or locally.
Stars: ✭ 510 (+1357.14%)
Mutual labels:  backup, rsync
View All Similar Projects ➔

Adlibre Backup

A high performance snapshot based backup system for Linux and UNIX like operating systems.

Designed with system administrators in mind.

Utilises native BTRFS / ZFS filesystem features for seamless compression, deduplication and snapshoting of the backup pool.

The problem

Existing Rsync backup approaches (eg Rsnapshot / BackupPC) don't scale, are hard to monitor and maintain when used with dozens or hundreds of hosts.

They also don't elegantly handle ad hoc backups, nor do they facilitate quickly adding and removing hosts.

Our solution

Our solution is centralised and agentless, so there is nothing to consume resources on your hosts, and all configuration is managed on the backup server.

Utilises BTRFS / ZFS native filesystem snapshots, and per host filesystems for better performance, scalability and ease of management.

Backups can be run at anytime, with custom expiry and a short message so you know why the backup was taken.

Aims

  • High performance and scalability. (Benchmarked faster than other Rsync backup solutions. eg BackupPC, Rsnapshot etc.)
  • Simplicity: Simple text based configuration. And simple files-on-disk backup format.
  • Aims to be a paranoid system administrator's best friend.

Features

  • Agentless
  • Utilises BTRFS / ZFS filesystem features, eg snapshot, dedup and compression
  • Uses Rsync and SSH for transport
  • Integration with monitoring tools such as Nagios or Icinga using NSCA passive checks. Or easily integrate your own monitoring system by configuring a MONITOR_HANDLER script.
  • Centralised configuration and management - all configuration and scheduling is done on the backup server
  • Ad hoc annotated backups - allows for ad hoc backups with an explanation as to when or why the backup was taken and per backup retention periods
  • Per host backup, retention and quota policies
  • Per host configuration and logs stored with the snapshot
  • Utilise LVM snapshots for performing atomic backups of Linux systems. See atomic.sh (Work in progress).

Installation

High level installation is as follows:

  1. An operating system with BTRFS or ZFS support is required (eg FreeBSD or ZFS on Linux) and a dedicated storage pool.

  2. Check out the source code into the root of your /backup pool storage device and review ./conf/backup.conf. Modify as necessary to set your pool and filesystem options.

Detailed: Red Hat / CentOS / EL Installation and ZFS Usage Example

Create backup zpool with dedup and compression.

zpool create -f backup vdb
zfs set dedup=on backup
zfs set compression=gzip backup

Install Adlibre Backup into root of backup zpool.

yum -y install git
cd /backup && git clone git://github.com/adlibre/adlibre-backup.git .

Install NSCA Client (optional) for Nagios / Icinga integration

yum -y install nsca-client

Generate SSH Key, this is used for authentication.

ssh-keygen -t rsa -N "" -f ~root/.ssh/id_rsa

Add server.example.com host config and copy the SSH Key to host example.com

cd /backup && ./bin/add-host.sh example.com

Now run the backup

./bin/backup-runner.sh --all

The output

[root@zbackup backup]# ./bin/backup-runner.sh --all
Info: Begin backup run of hosts example.com
Info: Begining backup of example.com
Running: rsync -a --numeric-ids --hard-links --compress --delete-after --delete-excluded --fuzzy --exclude=/dev --exclude=/proc --exclude=/sys --exclude=/tmp --exclude=/var/tmp --exclude=/var/run --exclude=/selinux --exclude=/cgroups --exclude=lost+found [email protected]:'/' /backup/hosts/example.com/d/
Warning: NSCA Plugin not found.
Backup Successful. Runtime 1757 seconds.
Warning: NSCA Plugin not found.
Snapshot example.com@2013-06-14-15:12:39-1371186759 Created
Info: Completed backup of example.com

That's it.

Now if you want to schedule daily backups Add the following to your root crontab:

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
@daily /backup/bin/backup-runner.sh --all --comment "Backup Daily" && /backup/bin/prune.sh --all

Usage

Adding a host

./bin/add-host.sh <hostname>

Then customise the per host config in ./hosts/<hostname>/c/backup.conf and ssh options in ~/.ssh/config if required.

Removing a host (ZFS)

To immediately purge the host configuration and all backup data:

zfs umount zfs-pool-name/hosts/<hostname> && zfs destroy zfs-pool-name/hosts/<hostname>

To disable future backups and allow existing backups to expire in line with the retention policy set DISABLED=true in ./hosts/<hostname>/c/backup.conf. This is the preferred method for host removal as it allows the old backups to naturally expire.

Running an ad hoc backup of a single host

./bin/backup.sh <hostname> <annotation> <expiry-in-days>

Running a backup of all hosts

./bin/backup-runner.sh --all

or multiple hosts

./bin/backup-runner.sh <hostname> <hostname>...

Running scheduled backups with flexible retention

You can configure cron jobs to run regular backups with flexible retentions.

The following example keeps daily backups for 2 weeks, weekly backups for 6 months and monthly backups for a year.

# daily backups, kept for 2 weeks
00 2 2-31 * * test $(date +\%u) != 7 && /backup/bin/backup-runner.sh --all --comment "Daily backup" --expiry 14 && /backup/bin/prune.sh --all
# weekly backups, kept for 6 months
00 2 2-31 * * test $(date +\%u) = 7 && /backup/bin/backup-runner.sh --all --comment "Weekly backup" --expiry 180 && /backup/bin/prune.sh --all
# monthly backups, kept for a year
00 2 1 * * /backup/bin/backup-runner.sh --all --comment "Monthly backup" --expiry 365 && /backup/bin/prune.sh --all

Restoring (ZFS)

All backups are stored on disk in plain sight. To restore all you need to do is copy (or rsync) the files from the backup pool to your host.

To find a particular snapshot:

./bin/list-backups.sh <hostname>

eg:

backup-host# ./bin/list-backups.sh example.com
example.com 2012-10-25-23:35:19-1351168519 1352377190 successful "first backup"
example.com 2012-11-04-15:40:49-1352004049 1354418267 successful "before acme software upgrade"

The files are stored plainly within the ZFS snapshot:

backup-host# ls -lah /backup/hosts/example.com/.zfs/snapshot
total 3
dr-xr-xr-x  4 root  wheel     4B Nov 17 15:14 .
dr-xr-xr-x  4 root  wheel     4B Oct 16 20:18 ..
drwxr-xr-x  5 root  wheel     5B Oct 16 20:18 2012-10-25-23:35:19-1351168519
drwxr-xr-x  5 root  wheel     5B Oct 16 20:18 2012-11-04-15:40:49-1352004049

Just dive in and copy the files out of the snapshot:

cd /backup/hosts/example.com/.zfs/snapshot/ && \
rsync -aH --numeric-ids 2012-11-04-15:40:49-1352004049/d/ example.com:/restore-point/

Upgrading

Upgrading is just a matter of updating the code and merging in any local changes to the default configuration. If you're using git to deploy the code then git pull is usually sufficient.

However from time to time options and features do sometimes change. Please review any changes to the defaults in etc/backup.conf.

The on disk pool format is stable and no changes are required to support upgrades.

Status

This has been in production use for many years now and is stable.

See TODO and ISSUES for outstanding issues. And NOTES for development information.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].