Skip to content

CephFS backups

all the swarm's stack data lives on one cephFS, passed to the docker VMs with virtioFS. VM backups do not include it, so it gets its own backup to PBS.

ceph takes a snapshot of the cephFS every hour on the hour. a small container on the proxmox cluster backs up the newest snapshot to PBS at quarter past, and proxmox HA restarts the container on another node when its node goes down, so the backup doesn't depend on any one node.

what where
snapshots ceph's snap_schedule, fs docker, path /, hourly, keeping 24 hourly and 7 daily
backup a systemd timer in the container, hourly at :15
source the host's /mnt/pve/docker-cephFS, bind mounted read-only at /mnt/cephfs
destination datastore mnt-pbs, namespace Files, group host/cephfs
credential the API token cephFS@pbs!cephFS, with DatastoreBackup only

a run takes about 15 seconds. the first one had nothing to compare against, so it read all 10 GiB and took two minutes.

1. the snapshots

on any node:

ceph mgr module enable snap_schedule
until ceph mgr module ls | grep -Eq '^snap_schedule +on'; do sleep 2; done
ceph fs snap-schedule add / 1h --fs docker
ceph fs snap-schedule retention add / 24h7d --fs docker
ceph fs snap-schedule status / --fs docker
  • the ceph manager takes the snapshots, so no particular node has to be up
  • enabling a module restarts the manager, and snap-schedule fails with not enabled/loaded until it has loaded the module. the until line waits
  • --fs docker names the filesystem. i have three, and without it the module uses the first in the fs map, which isn't this one
  • 31 snapshots at a time is under the module's limit of 50 per directory, and mds_max_snaps_per_dir's default of 100
  • the snapshots are a quick local undo too: copy a file back out of .snap. they are on the same cluster, so they are not the backup

ceph documents its snapshots as asynchronous, with buffered data flushed lazily. that is fine for config files but not for a database, see databases.

2. the container

a privileged debian 13 container, on vDisks so that HA can start it on any node:

pct create <ctid> ISOs-Templates:vztmpl/debian-13-standard_13.6-1_amd64.tar.zst \
  --hostname cephfs-backup \
  --ostype debian \
  --unprivileged 0 \
  --features nesting=1 \
  --cores 2 \
  --memory 1024 \
  --swap 512 \
  --rootfs vDisks:8 \
  --net0 name=eth0,bridge=vmbr0,ip=dhcp,ip6=auto \
  --mp0 /mnt/pve/docker-cephFS,mp=/mnt/cephfs,ro=1,shared=1 \
  --hookscript ISOs-Templates:snippets/check-donotdelete-hook.sh \
  --timezone America/Los_Angeles \
  --onboot 1
ha-manager add ct:<ctid> --state started
  • privileged, because files on cephFS belong to lots of different uids. an unprivileged container shifts uids, so through a bind mount files owned by uids outside its mapping show up as nobody, and anything not world-readable is off limits
  • privileged means root in the container is root on the host. it runs nothing but the backup, and that is still better contained than a cron job on the node itself
  • ro=1 stops the container changing the data it backs up
  • shared=1 tells proxmox the path exists on every node, which lets the container move
  • nesting=1 lets debian 13's systemd set up its own mounts and service namespaces. without it the container boots with its network down
  • the template has to be the amd64 build. pveam available lists an arm64 one too, which won't start on these nodes
  • the container has no ceph client and no key of its own. ceph's own network is the thunderbolt mesh, and a guest reaches that through vmbr100, whose subnet is different on each node, so a container that moves would lose its route
  • a container doesn't live-migrate. HA stops it on one node and starts it on another

3. the start guard

a bind mount is set up when the container starts. if cephFS isn't mounted on the host at that moment, the container gets the empty directory under the mount point, and keeps it after cephFS mounts. the container has the docker VMs' hookscript, so it refuses to start instead.

the script checks again before each run. it stops if .snap is missing, if the newest snapshot is more than two hours old or dated more than a minute in the future, or if the snapshot has no .donotdelete marker.

4. the backup

proxmox-backup-client comes from Proxmox's client repository. the script, the timer and its unit go in the container, and the token goes in /root/.pbs-cephfs-token, mode 600:

pbs-client.sources: Proxmox's client-only repository, for debian 13, 5 lines

download pbs-client.sources

1
2
3
4
5
Types: deb
URIs: http://download.proxmox.com/debian/pbs-client
Suites: trixie
Components: main
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
cephfs-backup.sh: the backup, run by the timer, 102 lines, 6 notes

each in the code opens a note on that line. download cephfs-backup.sh

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
#!/bin/bash

set -euo pipefail

SRC=/mnt/cephfs
MAX_AGE=$((2 * 60 * 60))  # (1)!
MAX_SKEW=60  # (2)!

export PBS_REPOSITORY='cephFS@pbs!cephFS@pbs1.mydomain.com:mnt-pbs'
export PBS_PASSWORD_FILE=/root/.pbs-cephfs-token

GATUS_URL="http://192.168.1.45:8085"
GATUS_ENDPOINT="plumbing_cephfs-backup"
GATUS_TOKEN_FILE=/root/.gatus-cephfs-backup-token

log() { echo "cephfs-backup: $*"; }

gatus_report() {
  local ok=$1 err=${2:-}
  if [[ ! -r "$GATUS_TOKEN_FILE" ]]; then
    log "no gatus token at $GATUS_TOKEN_FILE: result not reported"
    return 0
  fi
  printf 'Authorization: Bearer %s\n' "$(cat "$GATUS_TOKEN_FILE")" \
    | curl -fsS -m 20 -o /dev/null -X POST -H @- -G \
        --data-urlencode "success=$ok" --data-urlencode "error=$err" \
        --data-urlencode "duration=${SECONDS}s" \
        "$GATUS_URL/api/v1/endpoints/$GATUS_ENDPOINT/external" \
    || log "could not report the result to gatus"
}

reported=0
tmpdb=""
on_exit() {
  local rc=$?
  [[ -z "$tmpdb" ]] || rm -f "$tmpdb"
  if (( rc != 0 && reported == 0 )); then
    gatus_report false "exit $rc"
  fi
}
trap on_exit EXIT  # (3)!

fail() {
  log "FAILED: $*"
  gatus_report false "$*"
  reported=1
  exit 1
}

[[ -d "$SRC/.snap" ]] || fail "$SRC/.snap not found: is cephFS mounted on this node?"

newest="" newest_ts=0  # (4)!
for dir in "$SRC"/.snap/scheduled-*; do
  [[ -d "$dir" ]] || continue
  s=${dir##*/scheduled-}                      # 2026-09-25-13_00_00_PDT
  when="${s:0:10} ${s:11:2}:${s:14:2}:${s:17:2} ${s:20}"
  if ! ts=$(date -d "$when" +%s 2>/dev/null); then
    log "skipping ${dir##*/}: its name is not a time"
    continue
  fi
  if (( ts > newest_ts )); then
    newest=$dir
    newest_ts=$ts
  fi
done

[[ -n "$newest" ]] || fail "no scheduled snapshot under $SRC/.snap"
age=$(( $(date +%s) - newest_ts ))
(( age >= -MAX_SKEW )) \
  || fail "newest snapshot ${newest##*/} is dated $(( -age / 60 )) minutes in the future: is the ceph-mgr's clock right?"
(( age <= MAX_AGE )) \
  || fail "newest snapshot ${newest##*/} is $(( age / 60 )) minutes old: is snap_schedule running?"
[[ -e "$newest/.donotdelete" ]] || fail "${newest##*/} has no .donotdelete: not the docker cephFS"

portainer_db="$newest/portainer_data/portainer.db"
portainer_problem=""  # (5)!
if [[ -e "$portainer_db" ]]; then
  if ! { tmpdb=$(mktemp /tmp/portainer.XXXXXX) && cp "$portainer_db" "$tmpdb"; }; then
    portainer_problem="could not copy portainer.db from ${newest##*/} to /tmp to check it"
  elif out=$(timeout 300 bbolt check "$tmpdb" 2>&1) && [[ "$(tail -n 1 <<<"$out")" == OK ]]; then
    log "portainer.db in ${newest##*/} passes bbolt check"
  else
    portainer_problem="portainer.db in ${newest##*/} fails bbolt check: $(grep -m 1 -v '^[[:space:]]' <<<"$out" | cut -c 1-200)"
  fi
  [[ -z "$portainer_problem" ]] || log "$portainer_problem"
  [[ -z "$tmpdb" ]] || rm -f "$tmpdb"
  tmpdb=""
else
  portainer_problem="${newest##*/} has no portainer_data/portainer.db"
  log "$portainer_problem"
fi

log "backing up ${newest##*/}"
proxmox-backup-client backup "cephfs.pxar:$newest" \
  --ns Files \
  --backup-id cephfs \
  --change-detection-mode=metadata  # (6)!

[[ -z "$portainer_problem" ]] || fail "backed up, but $portainer_problem"
gatus_report true
reported=1
log "done in ${SECONDS}s"
  1. the newest snapshot may be at most two hours old. older means the schedule has stopped, and backing up the same snapshot every hour would hide that.
  2. nor may it be dated more than a minute ahead. a snapshot from a clock that jumped forward would otherwise stay the newest, and hide every later one while the same state was backed up.
  3. an exit that has not reported yet reports a failure to gatus, including one set -e causes, such as proxmox-backup-client failing.
  4. snapshot names carry the ceph manager's local time and zone, such as scheduled-2026-09-25-23_00_00_UTC, so the loop compares them as times. in text order, the hour that repeats when DST ends sorts wrong.
  5. portainer's database is checked in the snapshot with bbolt check before the backup. a copy that fails is still backed up, and the run then reports a failure, see databases.
  6. --backup-id cephfs, above, keeps every run in one group, host/cephfs, whichever node the container is on. the marker sits here because a line that ends in \ must keep it last.
cephfs-backup.service: the unit the timer starts, 10 lines

download cephfs-backup.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
[Unit]
Description=Back up the newest cephFS snapshot to PBS
Wants=network-online.target
After=network-online.target
RequiresMountsFor=/mnt/cephfs

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/cephfs-backup
Nice=10
cephfs-backup.timer: hourly at quarter past, 9 lines, 1 note

each in the code opens a note on that line. download cephfs-backup.timer

1
2
3
4
5
6
7
8
9
[Unit]
Description=Hourly cephFS backup, 15 minutes after each scheduled snapshot

[Timer]
OnCalendar=*-*-* *:15:00
Persistent=true  # (1)!

[Install]
WantedBy=timers.target
  1. a run missed while the container was stopped or moving between nodes runs as soon as it is back.

the script checks portainer's database in the snapshot with bbolt check before backing it up, see databases. debian has no bbolt command, so build it in the container from source at a pinned version, then remove go:

apt-get -y install golang-go
env HOME=/root GOBIN=/usr/local/bin go install go.etcd.io/bbolt/cmd/bbolt@v1.4.3
apt-get -y purge golang-go golang-1.24-go golang-1.24-src golang-src
apt-get -y autoremove --purge
rm -rf /root/go /root/.cache/go-build
  • DatastoreBackup can back up and restore the backups its token owns, but not prune or delete them, so a compromised container cannot wipe the history. retention is the prune job on the PBS side
  • --change-detection-mode=metadata compares file metadata with the previous backup instead of re-reading everything
  • --ns Files keeps these apart from the VM backups in the same datastore
  • proxmox-backup-client reads only the first line of PBS_PASSWORD_FILE, so the newline at the end of the file is fine

checking a restore

a backup that ran is not a backup that restores. verify-restore, in the container, restores one top-level directory from the newest backup into /var/tmp and compares it with the snapshot that backup read: the contents, and the owner, group and mode of every entry. i run it before relying on a change to how the backup runs:

pct exec <ctid> -- /usr/local/sbin/verify-restore
pct exec <ctid> -- /usr/local/sbin/verify-restore <directory>
  • not /tmp: on debian 13 that's a tmpfs, which counts against the container's memory, so restoring a directory bigger than that gets the restore killed. /var/tmp is on the container's disk, and the script stops first if it lacks room
verify-restore.sh: restores one directory from the newest backup and compares it with the snapshot, 61 lines

download verify-restore.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
#!/bin/bash

set -euo pipefail

SRC=/mnt/cephfs
DIR=${1:-acme_synology}
WORK_BASE=/var/tmp

export PBS_REPOSITORY='cephFS@pbs!cephFS@pbs1.mydomain.com:mnt-pbs'
export PBS_PASSWORD_FILE=/root/.pbs-cephfs-token

snap=$(journalctl -u cephfs-backup.service -o cat --no-pager \
  | sed -n 's/^cephfs-backup: backing up //p' | tail -n 1)
if [[ -z "$snap" || ! -d "$SRC/.snap/$snap" ]]; then
  echo "verify-restore: no snapshot from the journal, or it has been pruned: '$snap'" >&2
  exit 1
fi
if [[ ! -d "$SRC/.snap/$snap/$DIR" ]]; then
  echo "verify-restore: $DIR is not a directory in $snap" >&2
  exit 1
fi

ts=$(proxmox-backup-client snapshot list host/cephfs --ns Files --output-format json \
  | python3 -c 'import json, sys; print(max(s["backup-time"] for s in json.load(sys.stdin)))')
backup="host/cephfs/$(date -u -d "@$ts" +%Y-%m-%dT%H:%M:%SZ)"
echo "backup   $backup"
echo "snapshot $snap"

need=$(du -sk "$SRC/.snap/$snap/$DIR" | cut -f1)
free=$(df -k --output=avail "$WORK_BASE" | tail -n 1 | tr -d ' ')
if (( need + need / 10 > free )); then
  echo "verify-restore: $DIR needs about $((need / 1024)) MiB, $WORK_BASE has $((free / 1024)) MiB free" >&2
  exit 1
fi

work=$(mktemp -d "$WORK_BASE/verify-restore.XXXXXX")
trap 'rm -rf "$work"' EXIT

if ! proxmox-backup-client catalog dump "$backup" --ns Files > "$work/catalog" 2>&1; then
  tail -n 5 "$work/catalog" >&2
  exit 1
fi
echo "entries in the backup:   $(grep -c '^[a-z] "\./' "$work/catalog")"
echo "entries in the snapshot: $(find "$SRC/.snap/$snap" | wc -l)"

proxmox-backup-client restore "$backup" cephfs.pxar "$work/restore" --ns Files --pattern "$DIR/**"

status=0
if diff -rq "$SRC/.snap/$snap/$DIR" "$work/restore/$DIR"; then
  echo "contents of $DIR: identical"
else
  status=1
fi
(cd "$SRC/.snap/$snap" && find "$DIR" -printf '%p %U:%G %m\n' | sort) > "$work/snapshot.list"
(cd "$work/restore" && find "$DIR" -printf '%p %U:%G %m\n' | sort) > "$work/restore.list"
if diff "$work/snapshot.list" "$work/restore.list"; then
  echo "owners and modes of $DIR: identical"
else
  status=1
fi
exit "$status"

still to do

  • an alert when it stops. the script reports each run to a gatus external endpoint, and gatus alerts when no report arrives within two hours. the endpoint and its token don't exist yet, so for now the script logs that and backs up anyway
  • the backups hold portainer's database, which has credentials in it. the copy that leaves the box is encrypted, see backups. the datastore on truenas isn't
  • putting a folder back on cephFS from PBS. the backup container mounts cephFS read-only, so a restore has to land somewhere else and be copied back

databases

a snapshot can catch a database's files mid-write, so each database also gets a copy that is consistent on its own. which copy, and how to restore each, is on databases.