How to effectively monitor regular backups.

Imagine the following scenario: you created a script in bash to create a backup of a production database, say, an online store. After creating the script and adding it to crontab, everything worked flawlessly. After some time, say a month, the database became corrupted, for example, due to the installation of a faulty plugin. At that moment, you want to retrieve an updated database backup from last night and discover that the last database backup is from two weeks ago. What happened? Everything was working fine.

This nightmare scenario is more common than you might think, and perhaps it has even affected you personally. Scripts added to crontab fail without warning, causing so-called "silent errors." They can be caused by a variety of reasons, such as a full disk, permission changes, network timeouts, expired credentials, or simply a typo after a "quick fix."

The Problem with Unmonitored Backups

Traditional cron jobs have a fundamental flaw: they only report an error when they fail to run. For example, your backup script might fail:

• Run successfully but exit with errors
• Exit but generate empty or corrupted files
• Run but take 10 times longer than expected (a sign of problems)
• Skip tables due to permission issues

Before you know it, your backup retention period might expire—leaving you without any valid backups.

Monitoring Backup Scripts

The solution is simple: your backup script should actively report its status to an external monitor. Here's how to integrate database backups with CronMonitor.

MySQL/MariaDB Backup Example

#!/bin/bash

MONITOR_URL="https://cronmonitor.app/api/ping/your-unique-id"
BACKUP_DIR="/backups/mysql"
DATE=$(date +%Y%m%d_%H%M%S)
DB_NAME="production"

# Signal start
curl -s "${MONITOR_URL}/start"

# Perform backup
mysqldump --single-transaction \
    --routines \
    --triggers \
    "$DB_NAME" | gzip > "${BACKUP_DIR}/${DB_NAME}_${DATE}.sql.gz"

# Check if backup was successful and file is not empty
if [ $? -eq 0 ] && [ -s "${BACKUP_DIR}/${DB_NAME}_${DATE}.sql.gz" ]; then
    # Clean old backups (keep last 7 days)
    find "$BACKUP_DIR" -name "*.sql.gz" -mtime +7 -delete

    # Signal success
    curl -s "${MONITOR_URL}/complete"
else
    # Signal failure
    curl -s "${MONITOR_URL}/fail"
    exit 1
fi

PostgreSQL Backup Example

#!/bin/bash

MONITOR_URL="https://cronmonitor.app/api/ping/your-unique-id"
BACKUP_DIR="/backups/postgres"
DATE=$(date +%Y%m%d_%H%M%S)
DB_NAME="production"

curl -s "${MONITOR_URL}/start"

# Use custom format for flexibility
pg_dump -Fc "$DB_NAME" > "${BACKUP_DIR}/${DB_NAME}_${DATE}.dump"

if [ $? -eq 0 ] && [ -s "${BACKUP_DIR}/${DB_NAME}_${DATE}.dump" ]; then
    # Verify backup integrity
    pg_restore --list "${BACKUP_DIR}/${DB_NAME}_${DATE}.dump" > /dev/null 2>&1

    if [ $? -eq 0 ]; then
        curl -s "${MONITOR_URL}/complete"
    else
        curl -s "${MONITOR_URL}/fail"
        exit 1
    fi
else
    curl -s "${MONITOR_URL}/fail"
    exit 1
fi

Multi-Database Backup with Size Reporting

For more comprehensive monitoring, you can report backup sizes:

#!/bin/bash

MONITOR_URL="https://cronmonitor.app/api/ping/your-unique-id"
BACKUP_DIR="/backups"
DATE=$(date +%Y%m%d)
DATABASES="app_production analytics users"

curl -s "${MONITOR_URL}/start"

FAILED=0
TOTAL_SIZE=0

for DB in $DATABASES; do
    mysqldump --single-transaction "$DB" | gzip > "${BACKUP_DIR}/${DB}_${DATE}.sql.gz"

    if [ $? -ne 0 ] || [ ! -s "${BACKUP_DIR}/${DB}_${DATE}.sql.gz" ]; then
        FAILED=1
        echo "Backup failed for: $DB"
    else
        SIZE=$(stat -f%z "${BACKUP_DIR}/${DB}_${DATE}.sql.gz" 2>/dev/null || stat -c%s "${BACKUP_DIR}/${DB}_${DATE}.sql.gz")
        TOTAL_SIZE=$((TOTAL_SIZE + SIZE))
    fi
done

if [ $FAILED -eq 0 ]; then
    # Report success with metadata
    curl -s "${MONITOR_URL}/complete?msg=Backed%20up%20${TOTAL_SIZE}%20bytes"
else
    curl -s "${MONITOR_URL}/fail"
    exit 1
fi

How to configure CronMonitor to monitor a cron job

1. Create a new monitor in CronMonitor for the "backup" job
2. Set a projected schedule (e.g., daily at 2:00 AM)
3. Configure a grace period—allow enough time for large databases
4. Configure alerts via email, Slack, or Discord

Key backup monitoring settings:

• Schedule: Match the server's cron schedule exactly
• Grace period: Set it longer than the longest projected backup time

Best Practices

1. Verify, don't assume

Always verify that the backup file exists and has content. An empty gzipped file is still a "successful" command.

2. Test restores regularly

Backups are only as good as their restores. Schedule periodic restore tests—and monitor them too.

3. Monitor backup duration

CronMonitor tracks how long each job takes. A sudden increase in backup time often indicates growing data volume or performance issues.

4. Store backups on an external drive/server

Monitoring should include a synchronization step outside the production database server.

# After local backup succeeds
rsync -az "${BACKUP_DIR}/" remote:/backups/ && \
    curl -s "${MONITOR_URL}/complete" || \
    curl -s "${MONITOR_URL}/fail"

5. Document recovery procedures

For example, if you receive an alert on Slack at night about a failed backup attempt, you'll need clear steps on what to do next, not a debugging session.

Conclusion

Database backups are the last line of defense against data loss. They deserve more than a cron job and the hope that everything works and executes correctly as planned. Active monitoring provides immediate information about the problem – while you still have time to resolve it.

Start monitoring your backup scripts today. Your future self (the one who doesn't have to explain data loss to a client) will thank you.