Rsync

Debian packages

The following Debian packages need to be installed:

rsync

References

How to setup an rsync server:

man rsyncd.conf

Configuration

To let /etc/init.d/rsync start the daemon, the following change needs to be made to /etc/default/rsync:

RSYNC_ENABLE=true

Create the file /etc/rsyncd.conf with the following content:

[backups]
path = /var/backups
read only = true
hosts allow = 127.0.0.1 192.168.0.0/16
exclude = lost+found *
include = boot/*** etc/*** git/*** home/*** ldap/*** mysql/*** usr.local/*** var.lib/*** var.samba/*** var.www/***

[mp3]
path = /var/samba/media/mp3
read only = true
hosts allow = 127.0.0.1 192.168.0.0/16
uid = patrick

[itunes]
path = /var/samba/media/iTunes Media
read only = true
hosts allow = 127.0.0.1 192.168.0.0/16
uid = patrick

[electricsheep]
path = /var/www/herzbube.ch/electricsheep
read only = false
hosts allow = 127.0.0.1 192.168.0.0/16

Discussion:

• Four modules named "backup", "mp3", "itunes" and "electricsheep" are created
• "backup" module
  • The module provides read-only access to the directory /var/backups
  • Only clients in the network 192.168.0.0/16 can access the module
  • No authentification is required
  • The file or directory "lost+found" is not served
  • All other files or directories are also not served, unless they are explicitly named on the "include" line
  • On the "include" line, triple asterisks are necessary to allow the content of the folder to be served; without an asterisk, only the directory but not its content is served; one or two asterisks are, for some unfathomable reason, not good syntax and the entire directory is not served
• "mp3" and "iTunes" modules
  • Most of the stuff above also applies to these modules
  • In addition, authentication is required for user patrick
• "electricsheep" module
  • This module provides read+write access

Server side

There are different modes how rsync can be run on the server side. At the moment, I am only interested in the Daemon mode, because that is what is required for my purposes:

• my client should run from a cron service on my Mac
• my client should not need to authenticate

If authentication is necessary and my client (run from cron) should still perform an automatic login, some SSH configuration is required: possibly the .ssh/config file, but I am not sure about this.

Client side

Important: Mac OS X ships with an ancient and outdated version of rsync (2.6.9) that attempts to support extended attributes with the -E option, added by Apple. This Apple contribution does not seem to work properly, for instance when I tried to backup my iPhoto Library into a sparse bundle located on a Samba network filesystem, the Apple-modified version of rsync always sync'ed all the files that had extended attributes, even if the files and attributes hadn't changed at all. On Mac OS X, install a recent version of rsync via Homebrew or a similar package manager.

Preliminary notes:

• The -a option is a shortcut that performs the sync in "archive" mode. -a implies a large number of other options such as -r for recursive sync and -p for preserving permissions. For details see the man page.
• The -a option omits some options for things that might be difficult to preserve, depending on the target filesystem type:
  • The -X option preserves extended attributes. On Mac OS X this is an important option because resource forks are stored in an extended attribute (xattr name "com.apple.ResourceFork"). The Finder is also known to store information in an extended attribute (xattr name "com.apple.FinderInfo"), and one other commonly seen xattr is "com.apple.quarantine" for files downloaded from the Internet.
  • The -A option preserves ACLs
  • The -H option preserves hard links
• The -z option compresses file data during the transfer. TODO: It is unclear whether compression occurs also if two local filesystem paths are synced.

To get a copy of an exported module:

# Files in the destination folder that don't exist in the exported module are not deleted
rsync -avzX pelargir::backup ~/Desktop/backup

# Also delete files in the destination folder that don't exist in the exported module
rsync -avzX --delete pelargir::backup ~/Desktop/backup

To list all available modules:

rsync pelargir::

To incrementally copy files to a writable module (files on the destination side are never deleted, even if they disappear on the source side):

rsync -tqz ~/Library/Application\ Support/ElectricSheep/*.mpg pelargir::electricsheep

Access from cron

0 21 * * *  rsync -aqzX --delete pelargir::backup ~/Desktop/backup

Advanced synchronization examples

This creates the folder ./dst/src and synchronizes its content with the source folder. Anything else that exists directly below ./dst remains untouched.

rsync -avzX --delete ./src ./dst

This synchronizes the content of the ./src folder into the ./dst folder. Anything that exists directly below ./dst and that is not in ./src is deleted. The difference to the previous example is the trailing slash after the source folder name.

rsync -avzX --delete ./src/ ./dst

This excludes the folder (or file) foo from the sync. The --delete-excluded option makes sure that the excluded folder (or file) is also removed from the destination folder - this may be necessary if a previous sync included the folder (or file).

rsync -avzX --delete --exclude=/foo --delete-excluded ./src/ ./dst

# IMPORTANT: The exclude pattern must include the "src" folder if the "src" folder is created within the destination folder.
rsync -avzX --delete --exclude=/src/foo --delete-excluded ./src ./dst