Practical DevOps
How to Set Up Automated Multicloud Backups Using S3 CLI, Rclone and Cyberduck
Prepare consistent backups, copy them to two independent S3 destinations, schedule the job and verify that a restore works.
1. Prepare a consistent source and private destinations
This Linux example assumes AWS CLI v2, rclone, cron and flock are installed. Use a dedicated backup account. Obtain Gloman’s actual HTTPS endpoint, signing region, bucket and credentials from your service configuration; the examples intentionally contain placeholders.
Prepare application-consistent database dumps and a matching file snapshot in /srv/backup/staged. Include a SHA256SUMS manifest generated from the backup files using relative paths, excluding the manifest itself. Freeze this directory for the entire copy and verification run. Copying a live database directory is not a consistent database backup.
Use private buckets, encryption at rest and HTTPS. Limit credentials to the required buckets and operations; keep deletion or retention-administration privileges separate where supported. Confirm versioning or Object Lock, retention and recovery permissions for both providers before relying on them.
2. Verify S3 access with the AWS CLI
Run aws configure --profile gloman-backup interactively as the backup account, using the supplied credentials and region. Restrict access to its configuration files. Set the variables below to your real endpoint, region and bucket; never put credentials in the command line.
export S3_ENDPOINT='https://REPLACE-WITH-ASSIGNED-ENDPOINT'
export S3_REGION='REPLACE-WITH-SIGNING-REGION'
export S3_BUCKET='REPLACE-WITH-BUCKET'
aws --profile gloman-backup --region "$S3_REGION" \
--endpoint-url "$S3_ENDPOINT" s3 ls "s3://$S3_BUCKET/"
aws --profile gloman-backup --region "$S3_REGION" \
--endpoint-url "$S3_ENDPOINT" s3 cp /srv/backup/staged/SHA256SUMS \
"s3://$S3_BUCKET/connection-test/SHA256SUMS" --dryrun
The listing checks read access; a dry run does not prove write access. After checking the destination, remove --dryrun for this small test upload and download it again to verify. If needed, configure addressing style and checksum options to match the provider’s documented S3 compatibility; keep certificate verification enabled.
3. Configure two independent rclone remotes
Run rclone config as the same backup account. Create an S3 remote named gloman with provider Other and the assigned endpoint, region and credentials. Create a second remote named secondary for another provider, with separate credentials and a private destination bucket. Use the provider-specific settings that rclone documents.
The script expects the default rclone configuration for that account. Protect it with filesystem permissions; obscured secrets in a configuration file are not a secret vault. If configuration encryption is enabled, arrange a protected, noninteractive unlock mechanism for the scheduler.
First run rclone copy /srv/backup/staged gloman:YOUR-BUCKET/preflight --dry-run and inspect the paths. Repeat for secondary. The scheduled workflow below uses copy, which does not delete destination-only files, and a unique run prefix. This creates full recovery sets and consumes storage accordingly.
4. Copy and verify both destinations
Save this as /home/backup/bin/upload-backups.sh, replace both bucket names and make it executable. Verify the rclone path with command -v rclone and adjust PATH if needed. The frozen staged directory and manifest must already exist; both destinations are attempted even if one fails.
Verification downloads the copied data for comparison, adding read traffic and potentially provider charges. This is deliberate for the example; design a verification schedule suitable for large archives. Neither copy nor check tests database recovery on its own.
#!/bin/bash
set -euo pipefail
export PATH=/usr/local/bin:/usr/bin:/bin
umask 077
source_dir=/srv/backup/staged
test -s "$source_dir/SHA256SUMS"
(cd "$source_dir" && sha256sum --check SHA256SUMS)
run_id="$(date -u +%Y%m%dT%H%M%SZ)-$(hostname -s)-$$"
status=0
for destination in \
'gloman:REPLACE-PRIMARY-BUCKET' \
'secondary:REPLACE-SECONDARY-BUCKET'; do
target="$destination/server-backups/$run_id"
if rclone copy "$source_dir" "$target" --immutable \
--retries 3 --log-level INFO &&
rclone check "$source_dir" "$target" --download --log-level INFO; then
printf 'Verified %s\n' "$target"
else
printf 'FAILED %s\n' "$target" >&2
status=1
fi
done
exit "$status"
5. Schedule the job and monitor completion
Create /home/backup/.local/state with access restricted to the backup account. After a successful manual run, add this line to that account’s crontab. It runs at 02:15 in the server’s configured timezone and prevents overlapping upload jobs.
Coordinate the snapshot producer with this job so it cannot alter the staged files during upload. Connect failures and missing successful runs to your monitoring system. A log file alone does not alert anyone; configure log rotation and monitor free space.
15 2 * * * /usr/bin/flock -n /home/backup/.local/state/s3-backup.lock /home/backup/bin/upload-backups.sh >> /home/backup/.local/state/s3-backup.log 2>&1
Set retention deliberately at each destination and account for older versions and incomplete multipart uploads. Test lifecycle rules on disposable data before applying them to recovery copies. Independent credentials and an immutable or offline copy reduce the chance that one compromised host can destroy every backup.
6. Restore with rclone or inspect with Cyberduck
Choose a verified run prefix and a new empty restore directory. Download one complete set, check its manifest, then import the database and files into an isolated test system. Repeat using the secondary provider and measure the time needed to recover the application.
rclone copy gloman:REPLACE-PRIMARY-BUCKET/server-backups/REPLACE-RUN-ID \
/srv/restore-test/REPLACE-RUN-ID
cd /srv/restore-test/REPLACE-RUN-ID
sha256sum --check SHA256SUMS
For a manual inspection, create an Amazon S3 bookmark in Cyberduck with the assigned endpoint and credentials, using the provider’s required profile or region settings. Browse the bucket and download a test object. Cyberduck is useful for interactive transfers; this guide uses cron and rclone for unattended automation. Record a successful application restore, not just an object listing.