BackupScripts
Jump to navigation
Jump to search
This page documents a couple of backup shell scripts that I am using to implement my backup solution.
Copy script
rsync
The script that creates a copy of a data set using rsync is
htb-mkbackupcopy.sh
I have made the script somewhat generic and added it to herzbube's toolbox, my personal collection of shell scripts. Here's a direct link (GitHub) to the latest version of the script.
The script uses rsync to copy a source folder to a destination folder. The source folder can be on a remote machine, the destination folder is either on a local filesystem, on a remote machine reachable via SSH, on a Samba share or on a disk image (which in turn can be located on a Samba share). Here's the script's usage text:
pi@raspberrypi1:~$ htb-mkbackupcopy.sh -h
htb-mkbackupcopy.sh [-h] [-i <pattern>] [-e <pattern>] [-s <samba-spec>] [-d <disk-image-path>] [-l <limit-spec>] [[user@]host:]source destination
-h: Print this usage text
-i <pattern>: An include pattern to be passed to rsync.
This parameter can be specified multiple times. The
order in which include/exclude patterns are specified
is important.
-e <pattern>: An exclude pattern to be passed to rsync.
This parameter can be specified multiple times. The
order in which include/exclude patterns are specified
is important.
[-s <samba-spec>]: Specification of a Samba share to
mount. The specification conforms to the usual syntax
used by "mount -t smbfs" (see man "mount_smbfs").
[-d <disk-image-path>]: The path to a disk image to
mount. If -s is also specified, the path specified
here must exist on the Samba share after it is
mounted. The disk image can be anything that can be
processed by hdiutil.
[-l <limit-spec>]: Bandwith limit specification. The
value is forwarded as-is to the rsync option --bwlimit.
Example values are: 2g (2 GB/s), 1.5m (1.5 MB/s),
3000k or just 3000 (3000 KB/s).
[[user@]host:]source: Specification of source folder.
Specification of remote source folder conforms to
the usual rsync / SSH syntax.
destination: Specification of destination folder. If
-s is specified, the path specified here must exist
on the Samba share after it is mounted. If -d is
specified (regardless of whether -s is also specified),
the path specified here must exist on the disk image
after it is mounted. If destination contains a
colon (":") or at ("@") character, it is assumed
to be a remote destination conforming to the usual
rsync / SSH syntax and the options -s and -d are not
allowed.
Exit codes:
0: No error
2: Aborted by signal (e.g. Ctrl+C)
3: Error during initialisation
4: Error while checking arguments
5: Error during main program
sftp
A second script that can create a copy of a data set using sftp is
htb-sftp-get.sh
As with the first copy script, I have made this second script somewhat generic and added it to herzbube's toolbox, my personal collection of shell scripts. Here's a direct link (GitHub) to the latest version of the script.
The script uses sftp and sshpass (only if passwordless authentication is not possible) to copy a source folder to a destination folder. The source folder must be on a remote machine, the destination folder must be on a local filesystem. Here's the script's usage text:
pi@raspberrypi1:~$ htb-sftp-get.sh -h htb-sftp-get.sh [-h] [-d <distinct-regex>] [-l <local-file-name-regex>] [-f <password-file-path>] [-e <password-envvar-name>] [user@]host:source-folder-path destination-folder-path -h: Print this usage text [-d <distinct-regex>]: Specification of a regular expression to extract that part from the names of the remote files which can be used to identify distinct sets of files from which only the one with the most recent modification timestamp should be transferred. If no regex is specified, then no such optimization occurs and all files in the remote folder are transferred. The regex must be an sed extended regex that contains at least one group. If more than one group is in the regex, only the first one is used. Back references are used to extract the file name part matched by the group. For instance, if the files foo-2025-03-23.zip, foo-2025-03-24.zip, foobar-2025-03-23.zip and foobar-2025-03-24.zip exist, then the regex "^([^-]+-).*$" can be used to match the distinct parts "foo-" and "foobar-". Note that the character "-" must be included in the distinction group, to avoid problems with the distinction logic which would otherwise occur because "foo" is a substring of "foobar". Only the files foo-2025-03-24.zip and bar-2025-03-24.zip will be transferred (based on the modification timestamp, not the date in the file name!). [-l <local-file-name-regex>]: Specification of a regular expression to extract parts from the names of the remote files to use to form the local file name. If this option is not specified, then transferred files are stored locally under the same name they have remotely. This option is only valid if -d is also specified. The regex must be an sed extended regex that contains at least two groups. If more than two groups are in the regex, only the first two are used. Back references are used to extract the file name parts matched by the groups. In the example given for the -d option, if the regex "^([^-]+).*(.zip)$" were used this would cause the local file names to be foo.zip and bar.zip. [-f <password-file-path>]: Path to a file that contains the password to use for authentication. If neither this nor -e is specified, then passwordless authentication is assumed. If specified this option is forwarded to sshpass. [-e <password-envvar-name>]: Name of an environment variable that contains the password to use for authentication. If neither this nor -f is specified, then passwordless authentication is assumed. If specified this option is forwarded to sshpass. [user@]host:source-path: Specification of remote host and remote source path, optionally with the name of the user to authenticate as. The specification conforms to the SFTP syntax. Read the SFTP man page for details of different SFTP behaviour when you specify a file or a folder. Specifically, the options -d and -l only work as expected if the remote source path is a folder. destination-folder-path: Specification of destination folder. The folder is created if it does not exist. Exit codes: 0: No error 2: Aborted by signal (e.g. Ctrl+C) 3: Error during initialisation 4: Error while checking arguments 5: Error during main program
Solnet-specific helper script
A third script that makes use of the previous two scripts is
backup-solnet.sh
Unlike the first two scripts, this script is not generic and is not part of herzbube's toolbox, my personal collection of shell scripts. Instead this script is very specifically geared towards fetching database dump files from my current hosting provider Solnet, and feeding those dump files into my backup solution.
The RaspberryPi's cron configuration (wiki link) requires the scrip to be stored in the bin folder of the user pi's home directory. The executable bit must be set.
$ ls -l ~/bin total 4 -rwxr-xr-x 1 pi pi 4080 May 14 14:44 backup-solnet.sh
Here is the script content:
#!/bin/bash
# Fetch database dumps from Solnet hosting server and feed them into the backup
# solution.
#
# The script expects a backup copy name (a non-empty string) as the first (and
# only) argument. The script aborts and exits with code 1 if anything goes
# wrong.
#
# Steps:
# - Create a temporary download folder with a name based on BACKUPCOPY_NAME.
# - Invoke htb-sftp-get.sh to transfer the database dump files via SFTP from
# the Solnet hosting server to the download folder. The dump file names are
# expected to have a specific format - see below for details.
# - Unzip each database dump file inside the download folder.
# - Invoke htb-mkbackupcopy.sh to feed the uncompressed dumps into the backup
# solution, using BACKUPCOPY_NAME. It's important that the uncompressed
# dumps are used so that de-duplication can take place.
# - Delete the temporary download folder.
SCRIPT_NAME="$(basename $0)"
USAGE="Usage: $SCRIPT_NAME <backup-copy-name>"
if test $# -eq 0; then
echo "Backup copy name not specified"
echo "$USAGE"
exit 1
fi
if test $# -gt 1; then
echo "Too many arguments"
echo "$USAGE"
exit 1
fi
BACKUPCOPY_NAME="$1"
if test -z "$BACKUPCOPY_NAME"; then
echo "Empty backup copy name specified"
exit 1
fi
DOWNLOAD_FOLDER=/tmp/$BACKUPCOPY_NAME.$$
TEMPORARY_FOLDER="$DOWNLOAD_FOLDER/tmp"
rm -rf $DOWNLOAD_FOLDER
if test $? -ne 0; then
echo "Failed to remove download folder: $DOWNLOAD_FOLDER"
exit 1
fi
# The regexes used in the following command are geared towards backup file names
# that have this format:
# <database-name>-<username>-<year>-<month>-<day>.zip
# Example:
# mediawikidb-sys57-2025-03-22.zip
# The assumption is that I will never create a database whose name contains a
# dash ("-") character.
/usr/local/htb/bin/htb-sftp-get.sh -f ~/.ssh/sftp-password.sftp.solnet.ch -d '^([^-]+-).*$' -l '^([^-]+).*(.zip)$' herzbubech405@sftp.solnet.ch:/var/mysql-backup $DOWNLOAD_FOLDER
if test $? -ne 0; then
echo "Failed to get backup files from remote server"
exit 1
fi
# The following block replaces each .zip file with the uncompressed database dump it contains.
# It operates under the assumption that each .zip file contains exactly one file.
pushd "$DOWNLOAD_FOLDER" >/dev/null
if test $? -ne 0; then
echo "Failed to change working directory to download folder: $DOWNLOAD_FOLDER"
exit 1
fi
mkdir "$TEMPORARY_FOLDER"
if test $? -ne 0; then
echo "Failed to create temporary folder: $TEMPORARY_FOLDER"
exit 1
fi
for DB_FILE_ZIPPED in *.zip; do
DB_FILE_BASENAME="$(echo "$DB_FILE_ZIPPED" | sed -e 's/.zip$//')"
if test $? -ne 0; then
echo "Failed to determine base file name: $DB_FILE_ZIPPED"
exit 1
fi
pushd "$TEMPORARY_FOLDER" >/dev/null
if test $? -ne 0; then
echo "Failed to change working directory to temporary folder: $TEMPORARY_FOLDER"
exit 1
fi
# -j = junk (=discard) paths
unzip -j "../$DB_FILE_ZIPPED" >/dev/null
if test $? -ne 0; then
echo "Failed to extract .zip archive: $DB_FILE_ZIPPED"
exit 1
fi
mv * "../${DB_FILE_BASENAME}.sql"
if test $? -ne 0; then
echo "Failed to move extracted file: $(ls -l)"
exit 1
fi
popd >/dev/null
if test $? -ne 0; then
echo "Failed to change working directory back to download folder after changing to temporary folder: $TEMPORARY_FOLDER"
exit 1
fi
rm "$DB_FILE_ZIPPED"
if test $? -ne 0; then
echo "Failed to remove .zip archive: $DB_FILE_ZIPPED"
exit 1
fi
done
rmdir "$TEMPORARY_FOLDER"
if test $? -ne 0; then
echo "Failed to remove temporary folder: $TEMPORARY_FOLDER"
exit 1
fi
popd >/dev/null
if test $? -ne 0; then
echo "Failed to change working directory back to original folder after changing to download folder: $DOWNLOAD_FOLDER"
exit 1
fi
/usr/local/htb/bin/htb-mkbackupcopy.sh $DOWNLOAD_FOLDER/ /mnt/backup-copy/$BACKUPCOPY_NAME/
if test $? -ne 0; then
echo "Failed to create copy of extracted files"
exit 1
fi
rm -rf $DOWNLOAD_FOLDER
if test $? -ne 0; then
echo "Failed to remove download folder: $DOWNLOAD_FOLDER"
exit 1
fi
exit 0
Snapshot script
The script that creates a snapshot of a data set is
htb-mkbackupsnapshot.sh
As with the copy script, I have made the snapshot script somewhat generic and added it to herzbube's toolbox. Here's a direct link (GitHub) to the latest version of the script.
The script uses bup to create a snapshot of a specified source folder. Both the source folder and the bup repository must be somewhere on a local file system. Here's the script's usage text:
pi@raspberrypi1:~$ htb-mkbackupsnapshot.sh -h htb-mkbackupsnapshot.sh [-h] source bup-repository -h: Print this usage text source: Specification of folder to snapshot. bup-repository: Specification of the bup repository folder. Exit codes: 0: No error 2: Aborted by signal (e.g. Ctrl+C) 3: Error during initialisation 4: Error while checking arguments 5: Error during main program