My TAR incremental backup strategy

Quick tip:

Here is my TAR incremental backup strategy.  I have been using rsync for a long time but recently ran into a number of permission and symlink issues rsync’ing from an Ubuntu guest to a Windows host.

A few comments on the bash script below:

1.    The do_backup() function is a common function and it does all the work of the backup.

2.    The do_backup() function has to take 4 parameters

a.    $1 is the source directory.  This is the directory you want to backup, OR, this is the directory that has stuff in it you want to backup.

b.    $2 is the destination directory.  This directory is relative to ${backupHome}, but still starts with the / character.  This is where you want the TAR balls to end up.

c.    $3 is a name for the backup.  This is used to create the names of the TAR balls and the SNAR indexes.

d.    $4 is a command to run inside the source directory ($1) which will be piped to the `tar` command for backup.

3.    This strategy makes a full backup (level 0) and a single incremental backup (level 1).  The level 1 backup is (correctly) overwritten each time a backup is performed.

4.    A full backup is performed if:

a.    A backup has never been done before

b.    The size of the incremental backup is greater than 10% of the size of the full backup.

Bash script

#!/bin/bash

backupHome=/mnt/hgfs/Backup

#

# First check to see that that backup directory exists.

# If it does, continue. Otherwise, exit with error code.

#

if [ ! -d "${backupHome}" ]; then

        echo "Directory ${backupHome} does NOT exist, exit."

        exit 1;

else

        echo "Directory ${backupHome} exists, continue."

fi

# do_backup "/etc/postfix" "/etc/postfix" "main" 'echo main.cf'

function do_backup {

    local backupSrc=$1

    local backupDst=${backupHome}${2}

    local backupNme=$3

    local backupDir=$4

    local fullSnarFile=${backupDst}/${backupNme}_full.snar

   local fullTarFile=${backupDst}/${backupNme}_full.tar.gz

    local level1SnarFile=${backupDst}/${backupNme}_level_01.snar

    local level1TarFile=${backupDst}/${backupNme}_level_01.tar.gz

    echo "Backup: ${backupNme} from: ${backupSrc} to: ${backupDst}"

    mkdir -p ${backupDst}

    cd ${backupSrc}

    # Check file percentages and see if full backup is needed

    if [ -f ${fullTarFile} ] && [ -f ${level1TarFile} ]

    then

        local fullSize=$(stat -c%s ${fullTarFile})

        echo "  Size of ${fullTarFile} is ${fullSize}"

        local level1Size=$(stat -c%s ${level1TarFile})

        echo "  Size of ${level1TarFile} is ${level1Size}"

        local percentChange=$(echo "scale=2; (${level1Size}/${fullSize}) * 100" | bc)

        percentChange=$(printf "%.0f" ${percentChange})

        echo "  Percent change is ${percentChange}"

        if [ ${percentChange} -gt 10 ]

        then

            echo "  Prepare for a new full backup"

            rm -f ${fullSnarFile}

            rm -f ${level1SnarFile}

            rm -f ${level1TarFile}

        fi

    fi

    # If file does not exist, perform full backup

    if [ ! -f ${fullSnarFile} ]

    then

        echo "  Performing full backup"

        eval ${backupDir} | tar --listed-incremental ${fullSnarFile} -cpzf ${fullTarFile} -T -

    else

        echo "  Performing level 1 backup"

        cp -f ${fullSnarFile} ${level1SnarFile}

        eval ${backupDir} | tar --listed-incremental ${level1SnarFile} -cpzf ${level1TarFile} -T -

    fi

}

Examples

Let’s take a look at a few usage examples.  For these examples, assume that ${backupHome} is /mnt/hgfs/Backup/ like in the script.

Example 1:

do_backup "/home/apache" "/home/apache" "apache" 'echo .'

Example 1 means you want to backup the /home/apache directory to the /mnt/hgfs/Backup/home/apache directory. The parameter “apache” means the following files will be created:

/mnt/hgfs/Backup/home/apache/apache_full.snar

/mnt/hgfs/Backup/home/apache/apache_full.tar.gz

/mnt/hgfs/Backup/home/apache/apache_level_01.snar

/mnt/hgfs/Backup/home/apache/apache_level_01.tar.gz

Finally, ‘echo .’ means the TAR ball will contain everything inside the /home/apache directory

Example 2:

do_backup "/etc/postfix" "/etc/postfix" "main" 'echo main.cf'

Example 2 means you want to backup the /etc/postfix directory to the /mnt/hgfs/Backup/etc/postfix directory. The parameter “main” means the following files will be created:

/mnt/hgfs/Backup/etc/postfix/main_full.snar

/mnt/hgfs/Backup/etc/postfix/main_full.tar.gz

/mnt/hgfs/Backup/etc/postfix/main_level_01.snar

/mnt/hgfs/Backup/etc/postfix/main_level_01.tar.gz

Finally, ‘echo main.cf’ means the TAR ball will ONLY contain the /etc/postfix/main.cf file!

Example 3:

do_backup "/home/michael" "/home/michael" "michael" 'find . -type f -name ".forward" -o -type d -name "cron" -o

-type d -name "Applications" -o -type d -name ".ssh"'

Example 3 means you want to backup the /home/michael directory to the /mnt/hgfs/Backup/home/michael directory. The parameter “michael” means the following files will be created:

/mnt/hgfs/Backup/home/michael/michael_full.snar

/mnt/hgfs/Backup/home/michael/michael_full.tar.gz

/mnt/hgfs/Backup/home/michael/michael_level_01.snar

/mnt/hgfs/Backup/home/michael/michael_level_01.tar.gz

Finally, 'find . -type f -name ".forward" -o -type d -name "cron" –o -type d -name "Applications" -o -type d -name ".ssh"' means the TAR ball will ONLY contain the following:

/home/michael/.forward

/home/michael/cron/

/home/michael/Applications/

/home/michael/.ssh/

Enjoy!

References

Whipp, Paul. (2010, December). Using tar for full and incremental backups. paulwhippconsulting.com. Retrieved April 4, 2016, from http://paulwhippconsulting.com/blog/using-tar-for-full-and-incremental-backups/

Reading MANIFEST.MF Values From a JAR File on your Class Path

Abstract
Recently, I was looking at building a simple "about" page for my newest application (See project Riviera, a database versioning source code management tool).  I started to build a properties file Maven would filter values into, but then I realized it was not necessary.  The JAR MANIFEST.MF file had almost all the metadata already.  So all I had to do was read the properties from MANIFEST.MF, which I found more difficult than I thought it should be. Here's my solution.

Code
Listing 1 shows the code.

Listing 1: Code to read MANIFEST.MF from JAR file on the class path

Attributes attributes;
try {
    URL jarURL 
        = this.getClass().getProtectionDomain().getCodeSource().getLocation();
    
    URI manifestUri 
        = new URI(String.format("jar:%s!/%s", jarURL, JarFile.MANIFEST_NAME));
    
    InputStream is 
        = manifestUri.toURL().openStream()
    
    Manifest manifest 
        = new Manifest(is);
    
    attributes 
        = manifest.getMainAttributes();            

    is.close();
} catch (Exception e) {
    attributes = new Attributes();
}

Let's take a look at this code in a bit more detail.

The URI object is the key to being able to read the MANIFEST.MF file from a JAR.  To read the contents of a JAR file, a properly formatted URI string looks like this:

"jar:file:/C:/path/to/file.jar!/META-INF/MANIFEST.MF"

The jar:file and ! parts are required in the URI.  The rest of the URI depends on the specific location of the JAR file on your file system and what file in the JAR you want to stream.  Knowing what file you want to read is a given, but how do you get the fully-qualified location of the JAR file?

It stands to reason that if the JVM is executing code from a class it got from a JAR file then the JVM should be able to tell you where that JAR file is located. After all, the JVM needs to know where the JAR file is in order to load the class!  Turns out its simple to find this information.  You get the fully-qualified location of the JAR file from a class's ProtectionDomain that comes from the JAR file.  Once you have the fully-qualified location, you can build a properly formatted URI string.  Then once you have a URI object, then the magic happens.

From the URI object you can get a URL object by using URI#toUrl().  Then after you have a URL object you get an InputStream to the resource using the URL#openStream() method. Under the covers, the URL object determines it needs a JarURLConnection to open the stream.

Once you have the InputStream, pass it along to the Manifest constructor and you're done!  Use Manifest#getMainAttributes() to get the Attributes and then use Attributes#getValue(key) to get the value you want.

Let's it!  Now you can read the the name/value attributes from the MANIFEST.MF file.

Enjoy!

References
Bozho. (2010, February 16). How to read a file from a jar file?. stackoverflow.com. Retrieved January 27, 2016 from http://stackoverflow.com/questions/2271926/how-to-read-a-file-from-a-jar-file.

GlassFish Jersey JAX-RS REST Service Client with Cookies and HTTP Proxy Example

Abstract
Working with JAX-RS, I found most people tend to focus on the server side of things.  But there is a client side as well which doesn't involve JavaScript.  I found it difficult finding client-side examples which involved some more advanced, yet very common requirements, such as how to use an HTTP proxy for your client and how to send cookies to the server.  This article consolidates my research into one example.  As I bring in more requirements, I'll update this example.

System Requirements
This example was developed and run using the following system setup. If yours is different, it may work but no guarantees.

• jdk1.8.0_65_x64
• NetBeans 8.1 version 6.3
• Maven 3.0.5 (Bundled with NetBeans)
• Jersey 2.22.1 (From GlassFish)

The rest of the software dependencies can be found in this article.

Download 

Git code example from GitHub

Maven 
One of the first challenges is trying to determine which version of JAX-RS you're working with. Jersey is the reference implementation of JAX-RS, and there are a lot of examples showing how to do things with Jersey.  But Jersey has gone through a lot of changes.  Version 1.x is "Sun Jersey" then version 2.x was completely repackaged as "GlassFish Jersey".  As Jersey has evolved, a lot of breaking changes have occurred, so examples may not work anymore.

This is my Maven dependency configuration for Jersey 2.22.1.

 
  org.glassfish.jersey.core
  jersey-client
  2.22.1
 
 
  org.glassfish.jersey.connectors
  jersey-apache-connector
  2.22.1
 

javax.ws.rs.client.Client 
To create a JAX-RS client, the first thing you need is an instance of javax.ws.rs.client.Client.  Getting one of these is not as easy as you might think. I probably ran across a half dozen different ways of getting a Client instance, so it's very confusing.  Listing 1 shows my ClientFactory.  Let's take a look at it in more detail.

Listing 1: ClientFactory

import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import org.glassfish.jersey.apache.connector.ApacheConnectorProvider;
import org.glassfish.jersey.client.ClientConfig;
import org.glassfish.jersey.client.ClientProperties;

public class ClientFactory {
    public static Client create() 
    {       
        System.out.printf("ClientFactory%n%n");
        
        ClientConfig config = new ClientConfig();
        {        
            config.property(ClientProperties.PROXY_URI, "http://localhost:7777");        
        }
        
        ApacheConnectorProvider provider = new ApacheConnectorProvider();
        {
            config.connectorProvider(new ApacheConnectorProvider());        
        }
        
        Client client 
            = ClientBuilder.newClient(config);
        
        return client;
    }
}

Let's take a look at line 23 first. ClientBuilder.newClient(config) is a standard way to create a Client object.  Line 23 only uses classes from the JAX-RS API.  The config object however, created on line 12 is a Jersey-specific implementation of a JAX-RS interface.  Line 14 is where we set a property in the config object to specify the HTTP Proxy.  This configuration, however, doesn't do anything by itself because the default JAX-RS implementation doesn't understand HTTP Proxies.  This is why in the pom.xml we have a dependency on jersey-apache-connector.  This allows the defaults to be overridden with Apache HttpClient.  Lines 17-20 create an ApacheConnectorProvider and put it in the config object.  Without the ApacheConnectorProvider, the HTTP Proxy won't be used.  Now that we have a Client object, Let's see how to use it.

main(String [] args) 
Getting a  Client object was the first step. Now let's use the  Client object to make a REST-ful service call.  The main() application in listing 2 does this.

Listing 2: A main(String [] args) application

import javax.ws.rs.client.Client;
import javax.ws.rs.client.WebTarget;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;

public class Main {
    public static void main(String[] args) throws Exception {
        Client client 
            = ClientFactory.create();
        
        WebTarget target = client
            .target("http://jsonplaceholder.typicode.com")
            .path("posts/1")
        ;
        
        Response response = target
            .request(MediaType.TEXT_HTML)
            .accept(MediaType.APPLICATION_JSON)
            .header("User-Agent", "Java/1.8.0_65)
            .cookie("name1","value1")
            .cookie("name2","value2")
            .get(Response.class)
        ;
        
        System.out.printf("Response: %s%n%n", response);        
        System.out.printf("AllowdMethods: %s%n%n", response.getAllowedMethods());
        System.out.printf("Class: %s%n%n", response.getClass());
        System.out.printf("Cookies: %s%n%n", response.getCookies());
        System.out.printf("Date: %s%n%n", response.getDate());
        System.out.printf("Entity: %s%n%n", response.getEntity());
        System.out.printf("EntityTag: %s%n%n", response.getEntityTag());
        System.out.printf("Headers: %s%n%n", response.getHeaders());
        System.out.printf("Language: %s%n%n", response.getLanguage());
        System.out.printf("LastModified: %s%n%n", response.getLastModified());
        System.out.printf("Length: %s%n%n", response.getLength());
        System.out.printf("Links: %s%n%n", response.getLinks());
        System.out.printf("Location: %s%n%n", response.getLocation());
        System.out.printf("MediaType: %s%n%n", response.getMediaType());
        System.out.printf("Metadata: %s%n%n", response.getMetadata());
        System.out.printf("Status: %s%n%n", response.getStatus());
        System.out.printf("StatusInfo: %s%n%n", response.getStatusInfo());
        System.out.printf("StringHeaders: %s%n%n", response.getStringHeaders());
        System.out.printf("hasEntity: %b%n%n", response.hasEntity());
        System.out.printf("readEntity(String): %s%n%n", response.readEntity(String.class));
    }
}

On lines 8-9 I use my custom ClientFactory to get a Client object.  From the Client object, the next step is to get a WebTarget.  Lines 11-14 show this.  The target() method call is typically the base URL and the path() method call adds path information to the base.  The WebTarget is then used to make the call and get the response, and this can be done in many different ways.  In my example, I make the call and get the response on lines 16-23.  First it uses the request() method to configure what type of data the request is being sent to the server, then the accept() method configures what type of data the server will be sending back to the client.  The header() method is then used to override the default value for "User-Agent". Proxies are usually sensitive to the "User-Agent" header and will reject requests from JAX-RS clients using the default value.  So change the "User-Agent" value to something your proxy will accept.  Next, the cookie() method calls add cookies to the request.  Finally the get(Response.class) method call makes an HTTP GET call to the server and wraps the response from the server into a Response object.  Fun!

The rest of the code is just logging what's in the Response object.

That's it, enjoy!

References
NGloom. (2015, May 15). How to add a http proxy for Jersey2 Client. stackoverflow.com. Retrieved December 3, 2015 from http://stackoverflow.com/questions/18942648/how-to-add-a-http-proxy-for-jersey2-client.

theotherian. (2013, August 11). setting up jersey client 2.0 to use httpclient, timeouts, and max connections. theotherian.com. Retrieved December 3, 2015 from http://www.theotherian.com/2013/08/jersey-client-2.0-httpclient-timeouts-max-connections.html.