My TAR incremental backup strategy

Quick tip:

Here is my TAR incremental backup strategy.  I have been using rsync for a long time but recently ran into a number of permission and symlink issues rsync’ing from an Ubuntu guest to a Windows host.

A few comments on the bash script below:

1.    The do_backup() function is a common function and it does all the work of the backup.

2.    The do_backup() function has to take 4 parameters

a.    $1 is the source directory.  This is the directory you want to backup, OR, this is the directory that has stuff in it you want to backup.

b.    $2 is the destination directory.  This directory is relative to ${backupHome}, but still starts with the / character.  This is where you want the TAR balls to end up.

c.    $3 is a name for the backup.  This is used to create the names of the TAR balls and the SNAR indexes.

d.    $4 is a command to run inside the source directory ($1) which will be piped to the `tar` command for backup.

3.    This strategy makes a full backup (level 0) and a single incremental backup (level 1).  The level 1 backup is (correctly) overwritten each time a backup is performed.

4.    A full backup is performed if:

a.    A backup has never been done before

b.    The size of the incremental backup is greater than 10% of the size of the full backup.

Bash script

#!/bin/bash

backupHome=/mnt/hgfs/Backup

#

# First check to see that that backup directory exists.

# If it does, continue. Otherwise, exit with error code.

#

if [ ! -d "${backupHome}" ]; then

        echo "Directory ${backupHome} does NOT exist, exit."

        exit 1;

else

        echo "Directory ${backupHome} exists, continue."

fi

# do_backup "/etc/postfix" "/etc/postfix" "main" 'echo main.cf'

function do_backup {

    local backupSrc=$1

    local backupDst=${backupHome}${2}

    local backupNme=$3

    local backupDir=$4

    local fullSnarFile=${backupDst}/${backupNme}_full.snar

   local fullTarFile=${backupDst}/${backupNme}_full.tar.gz

    local level1SnarFile=${backupDst}/${backupNme}_level_01.snar

    local level1TarFile=${backupDst}/${backupNme}_level_01.tar.gz

    echo "Backup: ${backupNme} from: ${backupSrc} to: ${backupDst}"

    mkdir -p ${backupDst}

    cd ${backupSrc}

    # Check file percentages and see if full backup is needed

    if [ -f ${fullTarFile} ] && [ -f ${level1TarFile} ]

    then

        local fullSize=$(stat -c%s ${fullTarFile})

        echo "  Size of ${fullTarFile} is ${fullSize}"

        local level1Size=$(stat -c%s ${level1TarFile})

        echo "  Size of ${level1TarFile} is ${level1Size}"

        local percentChange=$(echo "scale=2; (${level1Size}/${fullSize}) * 100" | bc)

        percentChange=$(printf "%.0f" ${percentChange})

        echo "  Percent change is ${percentChange}"

        if [ ${percentChange} -gt 10 ]

        then

            echo "  Prepare for a new full backup"

            rm -f ${fullSnarFile}

            rm -f ${level1SnarFile}

            rm -f ${level1TarFile}

        fi

    fi

    # If file does not exist, perform full backup

    if [ ! -f ${fullSnarFile} ]

    then

        echo "  Performing full backup"

        eval ${backupDir} | tar --listed-incremental ${fullSnarFile} -cpzf ${fullTarFile} -T -

    else

        echo "  Performing level 1 backup"

        cp -f ${fullSnarFile} ${level1SnarFile}

        eval ${backupDir} | tar --listed-incremental ${level1SnarFile} -cpzf ${level1TarFile} -T -

    fi

}

Examples

Let’s take a look at a few usage examples.  For these examples, assume that ${backupHome} is /mnt/hgfs/Backup/ like in the script.

Example 1:

do_backup "/home/apache" "/home/apache" "apache" 'echo .'

Example 1 means you want to backup the /home/apache directory to the /mnt/hgfs/Backup/home/apache directory. The parameter “apache” means the following files will be created:

/mnt/hgfs/Backup/home/apache/apache_full.snar

/mnt/hgfs/Backup/home/apache/apache_full.tar.gz

/mnt/hgfs/Backup/home/apache/apache_level_01.snar

/mnt/hgfs/Backup/home/apache/apache_level_01.tar.gz

Finally, ‘echo .’ means the TAR ball will contain everything inside the /home/apache directory

Example 2:

do_backup "/etc/postfix" "/etc/postfix" "main" 'echo main.cf'

Example 2 means you want to backup the /etc/postfix directory to the /mnt/hgfs/Backup/etc/postfix directory. The parameter “main” means the following files will be created:

/mnt/hgfs/Backup/etc/postfix/main_full.snar

/mnt/hgfs/Backup/etc/postfix/main_full.tar.gz

/mnt/hgfs/Backup/etc/postfix/main_level_01.snar

/mnt/hgfs/Backup/etc/postfix/main_level_01.tar.gz

Finally, ‘echo main.cf’ means the TAR ball will ONLY contain the /etc/postfix/main.cf file!

Example 3:

do_backup "/home/michael" "/home/michael" "michael" 'find . -type f -name ".forward" -o -type d -name "cron" -o

-type d -name "Applications" -o -type d -name ".ssh"'

Example 3 means you want to backup the /home/michael directory to the /mnt/hgfs/Backup/home/michael directory. The parameter “michael” means the following files will be created:

/mnt/hgfs/Backup/home/michael/michael_full.snar

/mnt/hgfs/Backup/home/michael/michael_full.tar.gz

/mnt/hgfs/Backup/home/michael/michael_level_01.snar

/mnt/hgfs/Backup/home/michael/michael_level_01.tar.gz

Finally, 'find . -type f -name ".forward" -o -type d -name "cron" –o -type d -name "Applications" -o -type d -name ".ssh"' means the TAR ball will ONLY contain the following:

/home/michael/.forward

/home/michael/cron/

/home/michael/Applications/

/home/michael/.ssh/

Enjoy!

References

Whipp, Paul. (2010, December). Using tar for full and incremental backups. paulwhippconsulting.com. Retrieved April 4, 2016, from http://paulwhippconsulting.com/blog/using-tar-for-full-and-incremental-backups/