Tuesday, April 12, 2016

My TAR incremental backup strategy

Quick tip:

Here is my TAR incremental backup strategy.  I have been using rsync for a long time but recently ran into a number of permission and symlink issues rsync’ing from an Ubuntu guest to a Windows host.

A few comments on the bash script below:

1.    The do_backup() function is a common function and it does all the work of the backup.
2.    The do_backup() function has to take 4 parameters
a.    $1 is the source directory.  This is the directory you want to backup, OR, this is the directory that has stuff in it you want to backup.
b.    $2 is the destination directory.  This directory is relative to ${backupHome}, but still starts with the / character.  This is where you want the TAR balls to end up.
c.    $3 is a name for the backup.  This is used to create the names of the TAR balls and the SNAR indexes.
d.    $4 is a command to run inside the source directory ($1) which will be piped to the `tar` command for backup.
3.    This strategy makes a full backup (level 0) and a single incremental backup (level 1).  The level 1 backup is (correctly) overwritten each time a backup is performed.
4.    A full backup is performed if:
a.    A backup has never been done before
b.    The size of the incremental backup is greater than 10% of the size of the full backup.

Bash script

#!/bin/bash
backupHome=/mnt/hgfs/Backup

#
# First check to see that that backup directory exists.
# If it does, continue. Otherwise, exit with error code.
#
if [ ! -d "${backupHome}" ]; then
        echo "Directory ${backupHome} does NOT exist, exit."
        exit 1;
else
        echo "Directory ${backupHome} exists, continue."
fi

# do_backup "/etc/postfix" "/etc/postfix" "main" 'echo main.cf'
function do_backup {
    local backupSrc=$1
    local backupDst=${backupHome}${2}
    local backupNme=$3
    local backupDir=$4

    local fullSnarFile=${backupDst}/${backupNme}_full.snar
   local fullTarFile=${backupDst}/${backupNme}_full.tar.gz
    local level1SnarFile=${backupDst}/${backupNme}_level_01.snar
    local level1TarFile=${backupDst}/${backupNme}_level_01.tar.gz

    echo "Backup: ${backupNme} from: ${backupSrc} to: ${backupDst}"
    mkdir -p ${backupDst}
    cd ${backupSrc}

    # Check file percentages and see if full backup is needed
    if [ -f ${fullTarFile} ] && [ -f ${level1TarFile} ]
    then
        local fullSize=$(stat -c%s ${fullTarFile})
        echo "  Size of ${fullTarFile} is ${fullSize}"

        local level1Size=$(stat -c%s ${level1TarFile})
        echo "  Size of ${level1TarFile} is ${level1Size}"

        local percentChange=$(echo "scale=2; (${level1Size}/${fullSize}) * 100" | bc)
        percentChange=$(printf "%.0f" ${percentChange})
        echo "  Percent change is ${percentChange}"

        if [ ${percentChange} -gt 10 ]
        then
            echo "  Prepare for a new full backup"
            rm -f ${fullSnarFile}
            rm -f ${level1SnarFile}
            rm -f ${level1TarFile}
        fi
    fi

    # If file does not exist, perform full backup
    if [ ! -f ${fullSnarFile} ]
    then
        echo "  Performing full backup"
        eval ${backupDir} | tar --listed-incremental ${fullSnarFile} -cpzf ${fullTarFile} -T -
    else
        echo "  Performing level 1 backup"
        cp -f ${fullSnarFile} ${level1SnarFile}
        eval ${backupDir} | tar --listed-incremental ${level1SnarFile} -cpzf ${level1TarFile} -T -
    fi
}


Examples

Let’s take a look at a few usage examples.  For these examples, assume that ${backupHome} is /mnt/hgfs/Backup/ like in the script.

Example 1:

do_backup "/home/apache" "/home/apache" "apache" 'echo .'

Example 1 means you want to backup the /home/apache directory to the /mnt/hgfs/Backup/home/apache directory. The parameter “apache” means the following files will be created:

/mnt/hgfs/Backup/home/apache/apache_full.snar
/mnt/hgfs/Backup/home/apache/apache_full.tar.gz
/mnt/hgfs/Backup/home/apache/apache_level_01.snar
/mnt/hgfs/Backup/home/apache/apache_level_01.tar.gz

Finally, ‘echo .’ means the TAR ball will contain everything inside the /home/apache directory

Example 2:

do_backup "/etc/postfix" "/etc/postfix" "main" 'echo main.cf'

Example 2 means you want to backup the /etc/postfix directory to the /mnt/hgfs/Backup/etc/postfix directory. The parameter “main” means the following files will be created:

/mnt/hgfs/Backup/etc/postfix/main_full.snar
/mnt/hgfs/Backup/etc/postfix/main_full.tar.gz
/mnt/hgfs/Backup/etc/postfix/main_level_01.snar
/mnt/hgfs/Backup/etc/postfix/main_level_01.tar.gz

Finally, ‘echo main.cf’ means the TAR ball will ONLY contain the /etc/postfix/main.cf file!

Example 3:

do_backup "/home/michael" "/home/michael" "michael" 'find . -type f -name ".forward" -o -type d -name "cron" -o
-type d -name "Applications" -o -type d -name ".ssh"'

Example 3 means you want to backup the /home/michael directory to the /mnt/hgfs/Backup/home/michael directory. The parameter “michael” means the following files will be created:

/mnt/hgfs/Backup/home/michael/michael_full.snar
/mnt/hgfs/Backup/home/michael/michael_full.tar.gz
/mnt/hgfs/Backup/home/michael/michael_level_01.snar
/mnt/hgfs/Backup/home/michael/michael_level_01.tar.gz

Finally, 'find . -type f -name ".forward" -o -type d -name "cron" –o -type d -name "Applications" -o -type d -name ".ssh"' means the TAR ball will ONLY contain the following:

/home/michael/.forward
/home/michael/cron/
/home/michael/Applications/
/home/michael/.ssh/

Enjoy!

References

Whipp, Paul. (2010, December). Using tar for full and incremental backups. paulwhippconsulting.com. Retrieved April 4, 2016, from http://paulwhippconsulting.com/blog/using-tar-for-full-and-incremental-backups/