POSTS / 13 min read
If It's Not Backed Up, Did It Even Exist?
A quick dive into my server's backup strategy, with a pinch of humor and a lot of rsync magic.
Published Nov 23, 2024
When managing a server, having a robust and consistent backup strategy is essential. Losing your data is no laughing matter, and a proper backup plan ensures peace of mind. In this post, I’ll walk you through my 3-2-1 backup strategy and the custom script I use to automate my backups efficiently.
Table of Contents
- Backup Strategy: The 3-2-1 Rule
- My Tools: From
dd
to Rsync Magic - Cron Jobs for Scheduled Backups
- Notification Example
- Closing Thoughts
Backup Strategy: The 3-2-1 Rule
The 3-2-1 backup rule is a simple yet powerful guideline:
- 3 Copies of Your Data: This includes the primary data and two backups.
- 2 Different Storage Mediums: I use an external SSD and an HDD connected to my server. [1]
- 1 Copy Stored Offsite or Disconnected: I periodically copy data to a separate SSD that I plug into the server every two weeks.
Here’s how my setup works:
- Primary Disk (
SSD
): Stores all the active container data, bound to the server. - Backup Disk (
HDD_2
): A script runs nightly to copy essential data here, keeping it connected to the server. - Secondary Disk (
SYS_BK
): Every 2–3 weeks, I manually copy data here, and it remains disconnected from the server for additional safety.
This multi-layered approach ensures redundancy and minimizes risks like hardware failure or accidental deletions.
My Tools: From dd
to Rsync Magic
To periodically copy data to SYS_BK
, I use this simple command:
dd if=/dev/sda | gzip -c > /media/user/SYS_BK/backup.img
This creates a compressed image of my primary disk. While powerful, the dd
command should be used carefully—it can overwrite data if used incorrectly. Test on non-critical files before deploying it in your backup process.
For nightly automated backups, I use a modified version of Mike Rubel’s rsync snapshots guide. His method is efficient, leveraging incremental backups to save storage while ensuring data integrity.
Rsync Snapshot Script: Rotating Snapshots
Here is the script that runs the backup:
#!/bin/bash
# ----------------------------------------------------------------------
# makes handy rotating-filesystem-snapshot utility
# ----------------------------------------------------------------------
# slightly variation of http://www.mikerubel.org/computers/rsync_snapshots
# ----------------------------------------------------------------------
unset PATH;
# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;
RM=/bin/rm;
MV=/bin/mv;
CP=/bin/cp;
TOUCH=/usr/bin/touch;
RSYNC=/usr/bin/rsync;
LS=/bin/ls;
GREP=/usr/bin/grep;
WC=/usr/bin/wc;
MKDIR=/bin/mkdir;
TR=/usr/bin/tr;
HEAD=/usr/bin/head;
TAIL=/usr/bin/tail;
CURL=/usr/bin/curl;
DU=/usr/bin/du;
CUT=/usr/bin/cut;
FIND=/usr/bin/find;
DATE=/usr/bin/date;
DF=/usr/bin/df;
FINDMNT=/usr/bin/findmnt
# ------------- script setup ---------------------------------------------
MOUNT_DEVICE=/media/user/HDD_2;
SOURCE=/media/user/SSD/data/;
SNAPSHOT_RW=/media/user/HDD_2/snapshot;
EXCLUDES=/home/user/server-config/backup_exclude;
BACKUPS_NUM=4;
NTFY_ADDR="192.168.1.108:19492/backup";
# ------------- script parameters ----------------------------------------
VERBOSITY="";
BACKUP_NAME="backup";
NOTIFY_TIME="DELAY: 10s";
while [ $# -gt 0 ]; do
case "$1" in
-v|--verbosity)
VERBOSITY="v";
;;
-n=*|--name=*)
BACKUP_NAME="${1#*=}";
;;
-t=*|--time=*)
NOTIFY_TIME="${1#*=}";
;;
*)
$ECHO "Error: Invalid argument";
exit 1;
esac
shift
done
# ------------- functions ------------------------------------------------
function notify() {
if [[ $2 -eq 0 ]]; then
TAG="tada";
else
TAG="rotating_light";
fi;
$CURL -s \
-H "Title: Backup report" \
-H "Tags: $TAG" \
-H "$NOTIFY_TIME" \
-d "$1" \
$NTFY_ADDR &
}
function handle_rsync_error() {
case $1 in
1)
ERRORMESSAGE="Syntax or usage error"
;;
2)
ERRORMESSAGE="Protocol incompatibility"
;;
3)
ERRORMESSAGE="Errors selecting input/output files, dirs"
;;
4)
ERRORMESSAGE="Requested action not supported: an attempt was made to manipulate 64-bit files on a platform that cannot support them; or an option was specified that is supported by the client and not by the server."
;;
5)
ERRORMESSAGE="Error starting client-server protocol"
;;
6)
ERRORMESSAGE="Daemon unable to append to log-file"
;;
10)
ERRORMESSAGE="Error in socket I/O"
;;
11)
ERRORMESSAGE="Error in file I/O"
;;
12)
ERRORMESSAGE="Error in rsync protocol data stream"
;;
13)
ERRORMESSAGE="Errors with program diagnostics"
;;
14)
ERRORMESSAGE="Error in IPC code"
;;
20)
ERRORMESSAGE="Received SIGUSR1 or SIGINT"
;;
21)
ERRORMESSAGE="Some error returned by waitpid()"
;;
22)
ERRORMESSAGE="Error allocating core memory buffers"
;;
23)
ERRORMESSAGE="Partial transfer due to error"
;;
24)
ERRORMESSAGE="Partial transfer due to vanished source files"
;;
25)
ERRORMESSAGE="The --max-delete limit stopped deletions"
;;
30)
ERRORMESSAGE="Timeout in data send/receive"
;;
137)
ERRORMESSAGE="Fehlercode 137 :: No clear errorcode. Possible reason: The Rsync process might have crashed."
;;
*)
ERRORMESSAGE="An unidentifed error occured - maybe a new errorcode due to a new version of rsync"
;;
esac
if [[ $1 -ne 0 ]]; then
$ECHO "Error while performing the backup. Error: $ERRORMESSAGE";
notify "Error while performing the backup. Error: $ERRORMESSAGE" 1;
exit 2;
fi;
}
# ------------- the script itself ----------------------------------------
START=$($DATE +%s);
# make sure we're running as root
if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root. Exiting..."; exit 3; } fi
# exit if the backup directory is not available or it's not a mount point
if [[ ! -d $MOUNT_DEVICE ]]; then
$ECHO "Error: $MOUNT_DEVICE does not exist.";
notify "Error: $MOUNT_DEVICE does not exist." 1;
exit 4;
fi;
if [[ $($FINDMNT -o TARGET --noheadings --first-only --evaluate $MOUNT_DEVICE) != $MOUNT_DEVICE ]]; then
$ECHO "ERROR: $MOUNT_DEVICE is not a mount point.";
notify "ERROR: $MOUNT_DEVICE is not a mount point." 1;
exit 5;
fi
# if the snapshot directory does not exists, create it
if [[ ! -d $SNAPSHOT_RW ]]; then
$MKDIR -p $SNAPSHOT_RW;
fi;
# rotating snapshots
# step 1: delete the oldest snapshot, if it exists and maximum number of backup reached:
if [[ -d $SNAPSHOT_RW/$BACKUP_NAME.$BACKUPS_NUM && -d $SNAPSHOT_RW/$BACKUP_NAME.1 ]]; then
$RM -rf $SNAPSHOT_RW/$BACKUP_NAME.1;
fi;
# step 2: if maximum number of backup reached then
# shift the middle snapshot(s) back by one, if they exist
if [[ -d $SNAPSHOT_RW/$BACKUP_NAME.$BACKUPS_NUM ]]; then
for (( i=2; i<=$BACKUPS_NUM; i++ )); do
if [[ -d $SNAPSHOT_RW/$BACKUP_NAME.$i ]]; then
$MV $SNAPSHOT_RW/$BACKUP_NAME.$i $SNAPSHOT_RW/$BACKUP_NAME.$(($i-1));
fi;
done;
fi;
ACTUAL_BACKUPS_NUM=$($LS $SNAPSHOT_RW | $GREP "^$BACKUP_NAME" | $WC -l | $TR -d " ");
if [[ ! -d $SNAPSHOT_RW/$BACKUP_NAME.$(($ACTUAL_BACKUPS_NUM + 1)) ]]; then
$MKDIR -p $SNAPSHOT_RW/$BACKUP_NAME.$(($ACTUAL_BACKUPS_NUM + 1))$SOURCE;
fi;
# step 4: rsync from the system into the latest snapshot (notice that
# rsync behaves like cp --remove-destination by default, so the destination
# is unlinked first. If it were not so, this would copy over the other
# snapshot(s) too!
$RSYNC -a$VERBOSITY \
--delete \
--delete-excluded \
--exclude-from="$EXCLUDES" \
--link-dest="$SNAPSHOT_RW/$BACKUP_NAME.$(($ACTUAL_BACKUPS_NUM))$SOURCE" \
$SOURCE $SNAPSHOT_RW/$BACKUP_NAME.$(($ACTUAL_BACKUPS_NUM + 1))$SOURCE;
handle_rsync_error $?;
# step 5: update the mtime of latest backup to reflect the snapshot time
$TOUCH $SNAPSHOT_RW/$BACKUP_NAME.$(($ACTUAL_BACKUPS_NUM + 1));
TOTAL_SIZE_BACKUP_NAME_ONLY=$($DU -sch $SNAPSHOT_RW/$BACKUP_NAME* | $TAIL -n 1 | $CUT -f 1);
TOTAL_BACKUPS_BACKUP_NAME_ONLY=$($LS $SNAPSHOT_RW | $GREP "^$BACKUP_NAME" | $WC -l | $TR -d " ");
TOTAL_FILES_BACKUP_NAME_ONLY=$($FIND $SNAPSHOT_RW/$BACKUP_NAME* -type f | $WC -l | $TR -d " ");
TOTAL_SIZE=$($DU -sh $SNAPSHOT_RW | $CUT -f 1);
TOTAL_BACKUPS=$($LS $SNAPSHOT_RW | $WC -l | $TR -d " ");
TOTAL_FILES=$($FIND $SNAPSHOT_RW -type f | $WC -l | $TR -d " ");
USAGE_PERCENTAGE=$($DF $SNAPSHOT_RW | $GREP $MOUNT_DEVICE | $TR -s " " | $CUT -d " " -f 5);
END=$($DATE +%s);
ELAPSED=$(($END-$START));
notify "Backup done successfully in $ELAPSED seconds.
Total size of $TOTAL_BACKUPS_BACKUP_NAME_ONLY $BACKUP_NAME $BACKUP_NAME_BACKUP_NAME_ONLY backups: $TOTAL_SIZE_BACKUP_NAME_ONLY ($TOTAL_FILES_BACKUP_NAME_ONLY files).
Total size of $TOTAL_BACKUPS backups: $TOTAL_SIZE ($TOTAL_FILES files).
Percentage of disk usage: $USAGE_PERCENTAGE" 0;
Key Features of the Script:
- Rotating Snapshots:
- Oldest backups are deleted when the maximum number is reached.
- Existing snapshots are rotated to make room for the newest one.
- Incremental Backups:
- Uses rsync to copy only changed files, reducing storage usage and speeding up operations.
- Creates hard links to unchanged files for efficiency.
- Notifications:
- Sends a report on backup success or failure, including details like elapsed time, backup size, and disk usage.
- Exclusions:
- Avoids unnecessary files like caches, temp data, and logs using an exclusion file:
**/.DS_Store nextcloud/nextcloud-volume/data/user/files_trashbin qbittorrent/downloads jellyfin/cache jenkins/jenkins-data/logs
- Avoids unnecessary files like caches, temp data, and logs using an exclusion file:
Cron Jobs for Scheduled Backups
I use cron to manage automated backups. Here are my entries:
0 3 * * * /home/user/server-config/backup.sh --name=daily --time="At: 07:10"
0 4 * * 1 /home/user/server-config/backup.sh --name=weekly --time="At: 07:09"
This setup ensures:
- Daily Backups: Run at 3:00 AM, keeping the last 4 backups.
- Weekly Backups: Run at 4:00 AM every Monday, also retaining the last 4 backups.
With this configuration, I maintain a total of 8 snapshots (4 daily + 4 weekly).
Notification Example
I use ntfy for backup notifications. It sends instant feedback to my devices about the status of the backups. Here’s an example of the notification I receive after a successful backup:
This instant feedback keeps me informed of backup status, so I can address any issues immediately.
Closing Thoughts
This approach has served me well, providing both reliability and peace of mind. If you’re looking for a robust backup solution, I hope this post offers some inspiration.
[1] The original 3-2-1 rule recommends using two different types of media (e.g., SSD and HDD). However, this is no longer necessary. According to Backblaze:
Today, you don’t need to keep your data on two different types of media, but you do need to keep your data on two different devices.