Backup
To backup docker-scripts apps, it is usually enough to make a
backup of the directories /opt/docker-scripts/ and
/var/ds/, since the data of the apps are usually stored on the
host system (in subdirectories of /var/ds/). I also include
/root/ in the backup, since it may contain useful things (for
example maintenance scripts).
1. The backup script
The main backup script is /root/backup/backup.sh and it
looks like this:
backup/backup.sh#!/bin/bash -x
cd $(dirname $0)
MIRROR_DIR=${1:-/var/bak}
# mirror everything to a local dir
./mirror.sh $MIRROR_DIR
# backup the mirror directory to the storagebox
./borg.sh $MIRROR_DIR
# backup the incus setup
./incus-backup.sh
It has these main steps:
-
Mirror everything (
/root/,/var/ds/and/opt/docker-scripts/) to the directory/var/bak/. This is mainly done byrsync. -
Backup the mirror directory (
/var/bak/) to the StorageBox, using BorgBackup. -
Unrelated to the first two steps, backup also the setup and configuration of Incus, which is located at
/var/lib/incus/.
2. Run the script periodically
To run the backup script periodically, create a cron job like this:
cat <<EOF > /etc/cron.d/backup
30 3 * * * root bash -l -c "/root/backup/backup.sh &> /dev/null"
EOF
It runs the script every night at 3:30. The borg script takes care to keep only the latest 7 daily backups, the latest 4 weekly backups, and the latest 6 monthly backups. This makes sure that the size of the backups (on the StorageBox) does not grow without limits. Borg also uses deduplication and compression for storing the backups, and this helps to reduce further the size of the backup data.
We are using bash -l -c to run the command in a login
shell. This is because the default PATH variable in the cron
environment is limited and some commands inside the script will fail to
execute because they cannot be found. By running it with bash -l
-c we make sure that it is going to have the same environment
variables as when we execute it from the prompt.
We are also using &> /dev/null in order to ignore all the
stdout and stderr output. If the cron jobs produce some output,
the cron will try to notify us by email. Usually we would like to
avoid this.
3. The mirror script
The script that mirrors /root/, /var/ds/ and
/opt/docker-scripts/ to the directory /var/bak/ is
located at /root/backup/mirror.sh, and it looks like this:
This script is self-explanatory, easy to read and to understand what it is doing. Nevertheless, let’s discuss a few things about it.
-
The script does not only mirror the directories of the host, but also the directories
/root/,/var/ds/and/opt/docker-scripts/in Incus containers. We are assuming that in some Incus containers we have installed Docker and somedocker-scriptsapps. In this example script, there is only one such Incus container, namedvclab:local container_list="vclab"But if there are more, their names can be easily added to the list (separated by spaces).
-
Before mirroring
/var/ds/, we run the script/var/ds/_scripts/backup.sh. This script may run the commandds backupon some of the applications. This is needed only for those apps where the content of the app directory is not sufficient for making a successful restore of the application. Usually those are the apps where we would use ads remaketo update, instead of ads make(see also: Update). So, we make these necessarydocker-scriptsbackups, before making a mirror of the directory and a backup to the StorageBox. This backup script may look like this: -
We are using the option
--one-file-systemwith the commandrsync. This means that the data on any external file systems will not be mirrored, will be skipped. This is useful for cases like NextCloud or BBB, which may have big amounts of data, and we keep their data on an external storage. These data should be backed up separately. -
Before mirroring
/var/ds/which contains the applications and their data, we make sure to stop docker, which in turn will stop all the applications. If the data on the disk are constantly changing while we make the mirror, we may get a "mirror" with inconsistent data. -
For the Incus containers, we mount the filesystem of the container to a directory on the host, before mirroring, and unmount it afterwards.
-
For the
bbbcontainer, we mirror only the/root/directory, which contains also the docker applications (greenlight etc.). The data directory/var/bigbluebutton/is not included in this backup.
4. The borg script
This script makes a backup of the mirror directory (/var/bak/
by default) to the StorageBox. It is located at
/root/backup/borg.sh, and looks like this:
The main commands of this script are borg create, borg
prune and borg compact. The last part of the script just shows
the status of these commands.
The script makes sure to keep a backup only for the last 7 days, the last 4 weeks, and the last 6 months (17 backups in total). This prevents the size of the backups from growing without limits. Borg also uses deduplication, compression and compact to keep the size of the backup storage as small as possible.
Borg commands usually need to know on which borg repository they should work, and the passphrase that is used to encrypt that repository. These may be specified on the command line, or with environment variables. We are using environment variables because this is more convenient for a script:
export BORG_PASSPHRASE='XXXXXXXXXXXXXXXX'
export BORG_REPO='storagebox:backup/repo1'
The semicolon in the BORG_REPO tells borg to use SSH for
accessing the repo. The name of the SSH server (storagebox) is defined
in /root/.ssh/config and looks like this:
Host storagebox
HostName uXXXXXX.your-storagebox.de
User uXXXXXX
Port 23
IdentityFile /root/storagebox/key1
For some more details see: How to access the StorageBox with SSH keys
5. Initialize a borg repo
Before using the borg script, it is necessary to initialize a borg repository for the backup. It can be done like this:
pwgen 20
export BORG_PASSPHRASE='XXXXXXXXXXXXXXXXXXXX'
ssh storagebox mkdir -p backup/repo1
export BORG_REPO='storagebox:backup/repo1'
borg init -e repokey
borg key export > borg.repo1.key
| The repository data is totally inaccessible without the key and the passphrase. So, make sure to save them in a safe place, outside the server and outside the StorageBox. |
It is important to study at least the
Quick
Start tutorial of the BorgBackup, and to make some tests in order to
get familiar with borg.
|
6. Restore docker-scripts app
Let’s assume that we want to restore the application /var/ds/app1/.
-
First, we extract it from the Borg archive to a local directory:
export BORG_PASSPHRASE='XXXXXXXXXXXXXXXXXXXX' export BORG_REPO='storagebox:backup/repo1' borg list borg list ::<name-of-archive> borg list ::<name-of-archive> var/bak/host/var/ds/app1/ cd ~ borg extract ::<name-of-archive> var/bak/host/var/ds/app1/ ls var/bak/host/var/ds/app1/ -
Remove the current app (if installed):
cd /var/ds/app1/ ds remove cd .. rm -rf app1/ -
Copy/move the directory that was extracted from the backup archive:
cd /var/ds/ mv ~/var/bak/host/var/ds/app1/ . rm -rf ~/var/ -
Rebuild the app:
cd app1/ ds makeUsually this is sufficient for most of the apps, because the necessary settings, data, configurations etc. are already in the directory of the application.
In some cases, if there is a
.tgzbackup file, we may also need to do ads restore:ds restore backup*.tgz
7. Backup Incus
This is done by the script /root/backup/incus-backup.sh, which
is called by /root/backup/backup.sh. It looks like this:
The Incus directory is located at /var/lib/incus/. When we
make a mirror of this directory to the storagebox (with rsync),
we exclude the subdirectory disks/, which contains the storage
devices, where are stored the Incus containers etc. Instead, we are
making backups of the Incus containers separately.
We are also making a daily snapshot of all the containers. We are doing it in such a way that only the last 7 daily snapshots are preserved.
To backup the Incus containers, we use the command incus export
(with the option --optimized-storage), to create a compressed
archive of the filesystem of the container. This archive is stored on
a directory in the StorageBox, which is mounted with SSHFS. The
archives older than 4-5 days are removed. Not all the containers need
a full backup like this, because for some of them we actually backup
only the important configurations and data.