January 21, 2020

Java Mail Sent Over SSL

Blogger

Abstract

The purpose of this blog is to demonstrate how to use Java Mail to send an email using an SMTP server with an SSL connection.

Disclaimer

This post is solely informative. Critically think before using any information presented. Learn from it but ultimately make your own decisions at your own risk.

Requirements

I did all of the work for this post using the following major technologies. You may be able to do the same thing with different technologies or versions, but no guarantees.

  • NetBeans 11.2
  • Maven 3.3.9 (bundled with NetBeans)
  • Java 11 (zulu11.35.15-ca-jdk11.0.5-win_x64)
<dependency>
  <groupId>javax.mail</groupId>
  <artifactId>mail</artifactId>
  <version>1.4</version>
  <scope>test</scope>
</dependency>

Download

Visit my GitHub page https://github.com/mjremijan to see all of my open source projects. The code for this post is located at https://github.com/mjremijan/thoth-email in the https://github.com/mjremijan/thoth-email/tree/master/thoth-email-via-ssl module.

Properties

This example uses an smtp-ssl-yahoo.properties file to hold the SMTP server information. I used my personal Yahoo! account for testing, hence the use of the word yahoo in the name of the properties file. What’s important are the contents of the file, shown in Listing 1.

Listing 1 - Properties file

# This is the name of the SMTP host machine.
host=

# This is the port number of the SMTP host machine.
# The same host may support both SSL and TLS but on
# different ports. So make sure you get the SSL port.
port=

# This is what you use in the “username” field when
# you login. Typically this is the same as your email
# address, but this isn’t always the case.
username=

# This is what you use in the “password” field when
# you login. This value is CLEAR TEXT, so keep this
# properties file safe.
password=

# This is the email address you want for the
# email’s FROM field. Enter the value using
# the format shown below. Typically this is
# just your email address for the account.
from=FIRSTNAME LASTNAME <ADDRESS@EMAIL.COM>

# This is the email address you want for the
# email’s REPLY_TO field. Enter the value using
# the format shown below. Typically this is
# just your email address for the account. Also
# typically this is the same as `from` above.
# But be warned, if an email’s FROM and REPLY_TO 
# are different, that’s may be flagged as spam
# and never be delivered. So keep `from` and
# `reply` the same for initial testing
reply=FIRSTNAME LASTNAME <ADDRESS@EMAIL.COM>

# This is the email address you want to send
# the email to. For testing, it’s a good idea
# to send it to yourself first.
to=FIRSTNAME LASTNAME <ADDRESS@EMAIL.COM>

Now that you have a properties file, next let’s take a look at the code.

Code

This is a JUnit test demonstrating how to use Java Mail to send an email using an SMTP server with an SSL connection. Listing 2 shows the code.

NOTE For initial testing, always check your SPAM folder. A rule can always be added to deliver to your INBOX.

Listing 2 - Java Mail example

package org.thoth.email.via.ssl;

import java.net.InetAddress;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Properties;
import javax.mail.Authenticator;
import javax.mail.Message;
import javax.mail.PasswordAuthentication;
import javax.mail.Session;
import javax.mail.Transport;
import javax.mail.internet.InternetAddress;
import javax.mail.internet.MimeBodyPart;
import javax.mail.internet.MimeMessage;
import javax.mail.internet.MimeMultipart;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;

public class SslTest {

  public SslTest() {
  }

  protected String now, hostname;

  protected Properties yahoo;

  @BeforeEach
  public void setUp() throws Exception {
    now = new SimpleDateFormat("MM-dd-yyyy hh:mm:ss a").format(new Date());
    hostname = InetAddress.getLocalHost().getHostName();
    yahoo = new Properties();
    yahoo.load(this.getClass().getResourceAsStream("/smtp-ssl-yahoo.properties"));
  }

  @Test
  public void a_test() throws Exception {

    // Create MimeMultipart
    MimeMultipart content = new MimeMultipart("related");

    // html part
    {
      MimeBodyPart textPart = new MimeBodyPart();
      textPart.setText("<html><body>"
        + "<p>Time: "+now+"</p>"
        + "<p>From: "+hostname+"</p>"
        + "</body></html>"
        , "UTF8", "html");
      content.addBodyPart(textPart);
    }

    // properties
    Properties props = new Properties();
    {
      props.setProperty("mail.smtp.auth", "true");
      props.setProperty("mail.smtp.host", yahoo.getProperty("host"));
      props.setProperty("mail.smtp.socketFactory.port", yahoo.getProperty("port"));
      props.setProperty("mail.smtp.socketFactory.class", "javax.net.ssl.SSLSocketFactory");
    }

    Session smtp = null;
    {
      smtp = Session.getInstance(props, new Authenticator() {
        @Override
        protected PasswordAuthentication getPasswordAuthentication() {
          return new PasswordAuthentication(
              yahoo.getProperty("username")
            , yahoo.getProperty("password")
          );
        }
      });
      smtp.setDebug(true);
      smtp.setDebugOut(System.out);
    }


    MimeMessage m = new MimeMessage(smtp);
    {
      m.setRecipient(Message.RecipientType.TO, new InternetAddress(yahoo.getProperty("to")));
      m.setSubject("thoth-email SSL test " + now);

      InternetAddress from = null;
      {
        from = new InternetAddress(yahoo.getProperty("from"));
        from.setPersonal("Thoth Email");
        m.setFrom(from);
      }

      InternetAddress reply = null;
      {
        reply = new InternetAddress(yahoo.getProperty("reply"));
        m.setReplyTo(new InternetAddress[] {reply});
      }


      m.setContent(content);
    }

    Transport.send(m);
  }

}

Summary

The code for sending mail is not very hard. Successfully receiving an email without being flagged as SPAM is another matter. But if you follow this example, use a valid account, and don’t overuse it, you should be OK. This blog shows how to use Java Mail to send an email using an SMTP server with an SSL connection.

November 27, 2018

Postfix relay through Outlook (TLS)

Abstract

We take email for granted. It seems so easy. Just click the send button. But spam has made sending and receiving email a lot more complicated. This is especially true for people who like to run their own networks at home. If you are like me, you might have 1 or more physical machines running multiple virtual machines, not to mention multiple containers as well. All this network infrastructure needs to communicate with you somehow. After all, how do you keep tabs on whether or not your CRON jobs are successful? Email is the way. But sending email out from a home network without it being blocked at various points along the way can get complicated. We can go through the cost of buying a domain and email service, but who wants the cost? So how do we get the email through? This post explains how to configure Postfix to relay email through an Outlook account over TLS.

Disclaimer

This post is solely informative. Critically think before using any information presented. Learn from it but ultimately make your own decisions at your own risk.

Requirements

I did all of the work for this post using the following major technologies. You may be able to do the same thing with different technologies or versions, but no guarantees.

  • An Outlook account (outlook.com, live.com, etc.)
  • Ubuntu 18.04.1 LTS
  • Postfix 3.3.0

Install Postfix

I’m not going to go into a lot of explanation. I’m simply going to state the steps and configuration I did which got everything working.

First thing you need to do is install Postfix.

$ sudo apt-get update
$ sudo apt-get install postfix mailutils

NOTE For the Postfix installation, when it asks for the “System mail name” value, it should be the same as the name of the server.

Configure Postfix Authentication for Outlook

Configure Postfix to login to Outlook’s SMTP server. This configures the server name, port, account id, and clear text password.

# Create the password file
$ cd /etc/postfix/sasl
$ touch sasl_passwd_outlook
$ chmod 600 sasl_passwd_outlook

Now edit the sasl_passwd_outlook file using your favorite editor. Make it look like Listing 1, replacing ACCOUNT_NAME and CLEAR_TEXT_PASSWD appropriately.

Listing 1 - sasl_passwd_outlook

[smtp-mail.outlook.com]:587 ACCOUNT_NAME@outlook.com:CLEAR_TEXT_PASSWD

Now hash the sasl_passwd_outlook file into a Postfix .db file

$ postmap sasl_passwd_outlook

Configure Postfix Email Mapping for Outlook

If user mike attempts to send an email from a server with the name bluegreensky, the default Postfix FROM address will be mike@bluegreensky. This is not good because if you try to relay this email through Outlook, it will be blocked since the FROM address does not match the Outlook account. To get around this, configure a regular expression mapping file that will change all local email addresses (like mike@bluegreensky) to the Outlook account email address.

First, create a generic map file that’s empty

$ mkdir /etc/postfix/map
$ cd /etc/postfix/map
$ touch generic_map
$ chmod 600 generic_map
# Hash the file into a *.db file
$ postmap /etc/postfix/map/generic_map

Next, create an Outlook map file that will change local email address to the Outlook account email address.

$ mkdir /etc/postfix/map
$ cd /etc/postfix/map
$ touch regex_map_outlook
$ chmod 600 regex_map_outlook

Now edit the regex_map_outlook file using your favorite editor. Make it look like Listing 2, replacing HOSTNAME - just HOSTNAME, not @HOSTNAME…don’t lose the @ character - and ACCOUNT_NAME appropriately.

Listing 2 - regex_map_outlook

/.+@HOSTNAME/    ACCOUNT_NAME@outlook.com

Now hash the regex_map_outlook file into a Postfix .db file

$ postmap regex_map_outlook

Configure Postfix TLS for Outlook

TLS must be used to connect to Outlook SMTP servers. During Postfix installation, a main.cf is created. It must be edited

$ cd /etc/postfix

Now edit the main.cf file using your favorite editor. In Listing 3 are the values you must add or update in main.cf. For each of the name/value pairs below, search main.cf to see if it already exists. If so, use my value below. If not, add my value to the end of main.cf.

Listing 3 - main.cf

# Outlook
relayhost = [smtp-mail.outlook.com]:587
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_security_options =
smtp_sasl_password_maps = hash:/etc/postfix/sasl/sasl_passwd_outlook
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
smtp_tls_protocols = !SSLv2,!SSLv3,!TLSv1.1
smtp_tls_ciphers = medium
smtp_tls_mandatory_ciphers = medium
smtp_tls_security_level = may
smtp_generic_maps = hash:/etc/postfix/map/generic_map, regexp:/etc/postfix/map/regex_map_outlook

Restart Postfix

Restart Postfix so it picks up all the new configuration.

bash $ service postfix restart

Testing Postfix

If all goes well, Postfix restarted without any errors and is now configured to relay email from the server through the Outlook account. I typically test this with both mail and at.

# Test sending mail directly
$ echo "test email message" | mail -s "test email from server" some_email_address@someprovider.com
# Test sending mail through scheduler at
$ echo "echo \"gosh golly, it is AT\"" | at now

Do this testing, and see if you get the email your are expecting.

NOTE When testing with at, you will want to create a ~/.forward file with your email address so the results of the scheduled job are not delivered to your server’s local account’s inbox.

Summary

This post shows how to configure Postfix to relay email through an Outlook account over TLS. Sending email from a home network and have it not be blocked is tough. This configuration got me working. I hope it works for you.

November 26, 2018

Postfix relay through Yahoo! (SSL)

Abstract

We take email for granted. It seems so easy. Just click the send button. But spam has made sending and receiving email a lot more complicated. This is especially true for people who like to run their own networks at home. If you are like me, you might have 1 or more physical machines running multiple virtual machines, not to mention multiple containers as well. All this network infrastructure needs to communicate with you somehow. After all, how do you keep tabs on whether or not your CRON jobs are successful? Email is the way. But sending email out from a home network without it being blocked at various points along the way can get complicated. We can go through the cost of buying a domain and email service, but who wants the cost? So how do we get the email through? This post explains how to configure Postfix to relay email through a Yahoo! account over SSL.

Disclaimer

This post is solely informative. Critically think before using any information presented. Learn from it but ultimately make your own decisions at your own risk.

Requirements

I did all of the work for this post using the following major technologies. You may be able to do the same thing with different technologies or versions, but no guarantees.

  • A Yahoo! account
  • Ubuntu 18.04.1 LTS
  • Postfix 3.3.0

Install Postfix

I’m not going to go into a lot of explanation. I’m simply going to state the steps and configuration I did which got everything working.

First thing you need to do is install Postfix.

$ sudo apt-get update
$ sudo apt-get install postfix mailutils

NOTE For the Postfix installation, when it asks for the “System mail name” value, it should be the same as the name of the server.

Configure Yahoo! Account

The Yahoo! Account’s security setting must be set to allow password authentication. By default, it is not enabled. Go to the Yahoo! Account Security page and make sure “Password is enabled” is set as showing in Figure 1.

Figure 1 - Yahoo! Account Security

Yahoo! Account Security
Yahoo! Account Security

Configure Postfix Authentication for Yahoo!

Configure Postfix to login to Yahoo!’s SMTP server. This configures the server name, port, account id, and clear text password.

# Create the password file
$ cd /etc/postfix/sasl
$ touch sasl_passwd_yahoo
$ chmod 600 sasl_passwd_yahoo

Now edit the sasl_passwd_yahoo file using your favorite editor. Make it look like Listing 1, replacing ACCOUNT_NAME and CLEAR_TEXT_PASSWD appropriately.

Listing 1 - sasl_passwd_yahoo

[smtp.mail.yahoo.com]:465 ACCOUNT_NAME@yahoo.com:CLEAR_TEXT_PASSWD

Now hash the sasl_passwd_yahoo file into a Postfix .db file

$ postmap sasl_passwd_yahoo

Configure Postfix Email Mapping for Yahoo!

If user mike attempts to send an email from a server with the name bluegreensky, the default Postfix FROM address will be mike@bluegreensky. This is not good because if you try to relay this email through Yahoo!, it will be blocked since the FROM address does not match the Yahoo! account. To get around this, configure a regular expression mapping file that will change all local email addresses (like mike@bluegreensky) to the Yahoo! account email address.

First, create a generic map file that’s empty

$ mkdir /etc/postfix/map
$ cd /etc/postfix/map
$ touch generic_map
$ chmod 600 generic_map
# Hash the file into a *.db file
$ postmap /etc/postfix/map/generic_map

Next, create a yahoo map file that will change local email address to the Yahoo! account email address.

$ mkdir /etc/postfix/map
$ cd /etc/postfix/map
$ touch regex_map_yahoo 
$ chmod 600 regex_map_yahoo 

Now edit the regex_map_yahoo file using your favorite editor. Make it look like Listing 2, replacing HOSTNAME - just HOSTNAME, not @HOSTNAME…don’t lose the @ character - and ACCOUNT_NAME appropriately.

Listing 2 - regex_map_yahoo

/.+@HOSTNAME/    ACCOUNT_NAME@yahoo.com

Now hash the regex_map_yahoo file into a Postfix .db file

$ postmap regex_map_yahoo

Configure Postfix SSL for Yahoo!

SSL must be used to connect to Yahoo! SMTP servers. During Postfix installation, a main.cf is created. It must be edited

$ cd /etc/postfix

Now edit the main.cf file using your favorite editor. In Listing 3 are the values you must add or update in main.cf. For each of the name/value pairs below, search main.cf to see if it already exists. If so, use my value below. If not, add my value to the end of main.cf.

Listing 3 - main.cf

# Yahoo!
relayhost = [smtp.mail.yahoo.com]:465
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_security_options =
smtp_sasl_password_maps = hash:/etc/postfix/sasl/sasl_passwd_yahoo
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
smtp_tls_wrappermode = yes
smtp_tls_security_level = encrypt
smtp_generic_maps = hash:/etc/postfix/map/generic_map, regexp:/etc/postfix/map/regex_map_yahoo

Restart Postfix

Restart Postfix so it picks up all the new configuration.

bash $ service postfix restart

Testing Postfix

If all goes well, Postfix restarted without any errors and is now configured to relay email from the server through the Yahoo! account. I typically test this with both mail and at.

# Test sending mail directly
$ echo "test email message" | mail -s "test email from server" some_email_address@someprovider.com
# Test sending mail through scheduler at
$ echo "echo \"gosh golly, it is AT\"" | at now

Do this testing, and see if you get the email your are expecting.

NOTE When testing with at, you will want to create a ~/.forward file with your email address so the results of the scheduled job are not delivered to your server’s local account’s inbox.

Summary

This post shows how to configure Postfix to relay email through a Yahoo! account over SSL. Sending email from a home network and have it not be blocked is tough. This configuration got me working. I hope it works for you.

November 09, 2018

Derby Database Backup

Abstract

I have already posted a number of blogs about Derby:

This wasn’t intended to be a series. But over the years I’ve been using Derby more and more. Recently, I started using Derby as my database of choice for my Microservice architecture. These are personal-use applications, so Derby is more than sufficient. Even though these are personal-use applications, I require multiple servers with limited user permissions and - most importantly - backup. I’d hate to lose my data! The purpose of this post is to demonstrate how to backup a Derby database.

Disclaimer

This post is solely informative. Critically think before using any information presented. Learn from it but ultimately make your own decisions at your own risk.

Requirements

I did all of the work for this post using the following major technologies. You may be able to do the same thing with different technologies or versions, but no guarantees.

  • Apache Derby 10.14.2.0
  • OpenJDK 64-Bit Server VM Zulu11.1+23 (build 11-ea+22, mixed mode)

Download

There are no downloads with this blog post. The scripts are shown in full.

Derby System Utility

Backing up a Derby database is really quite simple. Derby has a built-in system utility for performing the backup. The utility is SYSCS_UTIL.SYSCS_BACKUP_DATABASE('/location/of/the/backup/'). When called, Derby will lock the database and perform the copy operation to the file system location you specify as the parameter to SYSCS_BACKUP_DATABASE. Now that we know the system utility to do the backup, let’s look at a bash script to automate it.

Backup Script

Listing 1 is a bash script which can be easily modified to backup any Derby database on any network server.

Listing 1 - derby-mydatabase-backup.sh

#!/bin/bash

# Define a bunch of variables which will be used within this script.
# The names of the variables should be self-explanatory.
DERBY_HOME=/opt/db-derby-10.14.2.0-bin/
NETWORK_SERVER_HOST=localhost
NETWORK_SERVER_PORT=1527
DATABASE_NAME=mydatabase
DATABASE_USER=sa
DATABASE_PASSWORD=abc123
JDBC_URL="jdbc:derby://$NETWORK_SERVER_HOST:$NETWORK_SERVER_PORT/$DATABASE_NAME"
BACKUP_DIRECTORY="/tmp/$DATABASE_NAME-backup/$NETWORK_SERVER_PORT"
BACKUP_SCRIPT="$BACKUP_DIRECTORY/backup.sql"

# Remove old backup if it exists. It is not a good idea to
# perform a backup on top of an existing backup.
rm -rf $BACKUP_DIRECTORY
mkdir -p $BACKUP_DIRECTORY
cd $BACKUP_DIRECTORY

# Use the `echo` command to dynamically create an SQL file.
# This SQL file will be used by Derby `ij` to connect to
# the database and perform the backup.
echo "connect '$JDBC_URL' user '$DATABASE_USER' password '$DATABASE_PASSWORD';" >> $BACKUP_SCRIPT
echo "CALL SYSCS_UTIL.SYSCS_BACKUP_DATABASE('$BACKUP_DIRECTORY');" >> $BACKUP_SCRIPT
echo "exit;" >> $BACKUP_SCRIPT

# Run the Derby `ij` application, passing it the SQL file
# that was just dynamically created. `ij` will read the 
# SQL file, executing its commands. This will then
# cause `ij` to connect to the database and call the 
# system utility to perform the backup.
$DERBY_HOME/bin/ij $BACKUP_SCRIPT

Let’s take a look at this script in more detail.

Lines 5–15 setup a number of variables used within the script. Some variables are used to set the values of other variables. There is nothing too complicated here. The names of the variables are self-explanatory.

Lines 17–19 is file system maintenance. It is not a good idea to perform a backup on top of an existing backup. So these lines remove an existing backup (if it exists) and creates a new, empty, backup directory.

Lines 24–26 are then responsible for creating the backup.sql script file. This script file contains the SQL commands to perform the backup. Line 24 is the connect command so Derby ij can connect to the database you want to backup. Line 25 is where the magic happens with a call to the SYSCS_BACKUP_DATABASE system utility. The location of the backup directory is passed as a parameter to the utility. When this SQL command is executed, Derby will lock the database and perform the backup. Line 26 is the exit command to exit ij.

Line 33 is then finally where everything happens. The Derby ij command is called with the location of the dynamically created backup.sql file passed to ij as a command-line parameter. When bash executes line 33, and if everything goes well, the Derby database will be backed up.

NOTE If you are running the Derby network server with a Java security policy, you may run into some problems with this script. the Java SecurityManager may prevent the network connection to the database or the SecurityManager my encounter permission problems writing to the backup directory.

Summary

Backing up a Derby database is pretty easy. Just call SYSCS_UTIL.SYSCS_BACKUP_DATABASE('/location/of/the/backup/').

References

Backing Up a Database. (2013, January 24). db.apache.org. Retrieved from https://db.apache.org/derby/docs/10.0/manuals/admin/hubprnt43.html.