Encrypted offsite backups with restic and S3-compatible storage

Install

restic is a single static Go binary. On Debian / Ubuntu:

sudo apt install restic
restic version

If the packaged version is older than the current upstream release (it usually lags by a couple of months), grab the latest binary directly:

restic self-update    # if you installed via the binary route
# — or —
curl -L https://github.com/restic/restic/releases/latest/download/restic_linux_amd64.bz2 \
    | bunzip2 > /usr/local/bin/restic
chmod +x /usr/local/bin/restic

Pick storage

Anything S3-compatible works the same way: Backblaze B2's S3 endpoint, Wasabi, Cloudflare R2, MinIO, Hetzner Object Storage, etc. Three pieces of information are needed: endpoint URL, access key, secret key. For B2 specifically there's a native B2 backend that doesn't need the S3 shim, but S3 mode works there too.

Create a bucket. Lifecycle/versioning settings on the bucket interact with restic in non-obvious ways — the safest default is no object-versioning, no lifecycle policy that deletes objects (restic manages its own retention with forget/prune). If you want immutability protection, use the provider's object-lock feature in compliance mode for a fixed retention period.

Initialize the repository

Set the four environment variables once, in a sourced file:

# /etc/restic/env  (chmod 600, root-only)
export RESTIC_REPOSITORY="s3:https://s3.us-west-002.backblazeb2.com/my-backups"
export RESTIC_PASSWORD_FILE="/etc/restic/password"
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."

Generate and store the password (this is what encrypts the repository — lose it, lose access to the backups):

openssl rand -base64 32 | sudo tee /etc/restic/password
sudo chmod 600 /etc/restic/password
# AND back it up out-of-band, somewhere not in the same trust domain as the server

Now init:

source /etc/restic/env
restic init

restic init writes a config object and a keys/ directory holding the scrypt-wrapped master key. Every chunk written afterward is encrypted with a per-repo data key, so re-keying the repo (adding/removing passwords) doesn't require re-encrypting the data.

First backup

source /etc/restic/env

restic backup \
    --tag nightly \
    --exclude-caches \
    --exclude="/var/lib/postgresql" \
    --exclude="**/node_modules" \
    /etc /home /var/www /srv

The first run uploads everything; subsequent runs upload only new content (chunk-level dedup with content-defined chunking). A snapshot is a small JSON metadata object naming the chunks that compose the tree; many snapshots that share chunks pay storage cost only once.

For database state, pipe a dump into restic rather than backing up the live data directory:

pg_dumpall -U postgres | restic backup --stdin --stdin-filename pg_dumpall.sql --tag db
mysqldump --all-databases --single-transaction \
    | restic backup --stdin --stdin-filename mysqldump.sql --tag db

Inspect and restore

restic snapshots
restic snapshots --tag nightly --host edge-01    # filter

restic ls latest
restic ls latest /etc/nginx                       # browse a snapshot tree

restic restore latest --target /tmp/restore --include /etc/nginx
restic restore <snapshot-id> --target /            # full restore overwriting current files

restic mount /mnt/restic exposes every snapshot as a directory tree under a FUSE mount — the fastest way to pull a single file out of a months-old snapshot without restoring anything else.

Forget & prune (the lifecycle)

By default, nothing in a restic repository is deleted — snapshots accumulate forever. Retention is a two-step process: forget drops snapshot references, prune garbage-collects unreachable chunks.

restic forget \
    --keep-daily 7 \
    --keep-weekly 4 \
    --keep-monthly 12 \
    --keep-yearly 3 \
    --prune

The --keep-* flags express the retention policy: 7 daily, 4 weekly, 12 monthly, 3 yearly snapshots, per host+tag combination. --prune at the end re-packs pack files to reclaim space from the deleted snapshots. Without --prune, forget only removes the snapshot pointers; storage is unchanged.

Integrity checks

Periodically verify the repository contents are intact and the metadata is consistent:

restic check                          # metadata + structural consistency
restic check --read-data-subset=5%    # also reads & verifies 5% of pack contents

A monthly --read-data-subset=10% rotates through the data over a year and catches bit-rot or storage-side corruption before a restore actually needs that data.

Nightly via systemd

Create /etc/systemd/system/restic-backup.service:

[Unit]
Description=Restic backup
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
EnvironmentFile=/etc/restic/env
ExecStart=/usr/bin/restic backup --tag nightly --exclude-caches /etc /home /var/www /srv
ExecStartPost=/usr/bin/restic forget --tag nightly --keep-daily 7 --keep-weekly 4 --keep-monthly 12
Nice=19
IOSchedulingClass=best-effort
IOSchedulingPriority=7

And /etc/systemd/system/restic-backup.timer:

[Unit]
Description=Run restic backup nightly

[Timer]
OnCalendar=*-*-* 03:17:00
RandomizedDelaySec=20min
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now restic-backup.timer
systemctl list-timers restic-backup.timer

Add a second timer for weekly restic prune and a monthly restic check --read-data-subset=10%.

The two things that go wrong

First: the password is irrecoverable if lost. There is no recovery key, no support hotline, no email-link reset. Back up /etc/restic/password and the env file to a separate trust domain — a password manager, a printed copy in a safe, a different cloud account.

Second: the bucket credentials are usually the same credentials used by the live system, which means a compromised machine can delete its own backups. Mitigations: a write-only IAM role for backups + a read-only role for restores; bucket-level object-lock for an immutability window; or restic's append-only mode with a separate "manager" credential that does forget/prune from elsewhere.