November 27, 2018

Postfix relay through Outlook (TLS)

Abstract

We take email for granted. It seems so easy. Just click the send button. But spam has made sending and receiving email a lot more complicated. This is especially true for people who like to run their own networks at home. If you are like me, you might have 1 or more physical machines running multiple virtual machines, not to mention multiple containers as well. All this network infrastructure needs to communicate with you somehow. After all, how do you keep tabs on whether or not your CRON jobs are successful? Email is the way. But sending email out from a home network without it being blocked at various points along the way can get complicated. We can go through the cost of buying a domain and email service, but who wants the cost? So how do we get the email through? This post explains how to configure Postfix to relay email through an Outlook account over TLS.

Disclaimer

This post is solely informative. Critically think before using any information presented. Learn from it but ultimately make your own decisions at your own risk.

Requirements

I did all of the work for this post using the following major technologies. You may be able to do the same thing with different technologies or versions, but no guarantees.

  • An Outlook account (outlook.com, live.com, etc.)
  • Ubuntu 18.04.1 LTS
  • Postfix 3.3.0

Install Postfix

I’m not going to go into a lot of explanation. I’m simply going to state the steps and configuration I did which got everything working.

First thing you need to do is install Postfix.

$ sudo apt-get update
$ sudo apt-get install postfix mailutils

NOTE For the Postfix installation, when it asks for the “System mail name” value, it should be the same as the name of the server.

Configure Postfix Authentication for Outlook

Configure Postfix to login to Outlook’s SMTP server. This configures the server name, port, account id, and clear text password.

# Create the password file
$ cd /etc/postfix/sasl
$ touch sasl_passwd_outlook
$ chmod 600 sasl_passwd_outlook

Now edit the sasl_passwd_outlook file using your favorite editor. Make it look like Listing 1, replacing ACCOUNT_NAME and CLEAR_TEXT_PASSWD appropriately.

Listing 1 - sasl_passwd_outlook

[smtp-mail.outlook.com]:587 ACCOUNT_NAME@outlook.com:CLEAR_TEXT_PASSWD

Now hash the sasl_passwd_outlook file into a Postfix .db file

$ postmap sasl_passwd_outlook

Configure Postfix Email Mapping for Outlook

If user mike attempts to send an email from a server with the name bluegreensky, the default Postfix FROM address will be mike@bluegreensky. This is not good because if you try to relay this email through Outlook, it will be blocked since the FROM address does not match the Outlook account. To get around this, configure a regular expression mapping file that will change all local email addresses (like mike@bluegreensky) to the Outlook account email address.

First, create a generic map file that’s empty

$ mkdir /etc/postfix/map
$ cd /etc/postfix/map
$ touch generic_map
$ chmod 600 generic_map
# Hash the file into a *.db file
$ postmap /etc/postfix/map/generic_map

Next, create an Outlook map file that will change local email address to the Outlook account email address.

$ mkdir /etc/postfix/map
$ cd /etc/postfix/map
$ touch regex_map_outlook
$ chmod 600 regex_map_outlook

Now edit the regex_map_outlook file using your favorite editor. Make it look like Listing 2, replacing HOSTNAME - just HOSTNAME, not @HOSTNAME…don’t lose the @ character - and ACCOUNT_NAME appropriately.

Listing 2 - regex_map_outlook

/.+@HOSTNAME/    ACCOUNT_NAME@outlook.com

Now hash the regex_map_outlook file into a Postfix .db file

$ postmap regex_map_outlook

Configure Postfix TLS for Outlook

TLS must be used to connect to Outlook SMTP servers. During Postfix installation, a main.cf is created. It must be edited

$ cd /etc/postfix

Now edit the main.cf file using your favorite editor. In Listing 3 are the values you must add or update in main.cf. For each of the name/value pairs below, search main.cf to see if it already exists. If so, use my value below. If not, add my value to the end of main.cf.

Listing 3 - main.cf

# Outlook
relayhost = [smtp-mail.outlook.com]:587
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_security_options =
smtp_sasl_password_maps = hash:/etc/postfix/sasl/sasl_passwd_outlook
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
smtp_tls_protocols = !SSLv2,!SSLv3,!TLSv1.1
smtp_tls_ciphers = medium
smtp_tls_mandatory_ciphers = medium
smtp_tls_security_level = may
smtp_generic_maps = hash:/etc/postfix/map/generic_map, regexp:/etc/postfix/map/regex_map_outlook

Restart Postfix

Restart Postfix so it picks up all the new configuration.

bash $ service postfix restart

Testing Postfix

If all goes well, Postfix restarted without any errors and is now configured to relay email from the server through the Outlook account. I typically test this with both mail and at.

# Test sending mail directly
$ echo "test email message" | mail -s "test email from server" some_email_address@someprovider.com
# Test sending mail through scheduler at
$ echo "echo \"gosh golly, it is AT\"" | at now

Do this testing, and see if you get the email your are expecting.

NOTE When testing with at, you will want to create a ~/.forward file with your email address so the results of the scheduled job are not delivered to your server’s local account’s inbox.

Summary

This post shows how to configure Postfix to relay email through an Outlook account over TLS. Sending email from a home network and have it not be blocked is tough. This configuration got me working. I hope it works for you.

November 26, 2018

Postfix relay through Yahoo! (SSL)

Abstract

We take email for granted. It seems so easy. Just click the send button. But spam has made sending and receiving email a lot more complicated. This is especially true for people who like to run their own networks at home. If you are like me, you might have 1 or more physical machines running multiple virtual machines, not to mention multiple containers as well. All this network infrastructure needs to communicate with you somehow. After all, how do you keep tabs on whether or not your CRON jobs are successful? Email is the way. But sending email out from a home network without it being blocked at various points along the way can get complicated. We can go through the cost of buying a domain and email service, but who wants the cost? So how do we get the email through? This post explains how to configure Postfix to relay email through a Yahoo! account over SSL.

Disclaimer

This post is solely informative. Critically think before using any information presented. Learn from it but ultimately make your own decisions at your own risk.

Requirements

I did all of the work for this post using the following major technologies. You may be able to do the same thing with different technologies or versions, but no guarantees.

  • A Yahoo! account
  • Ubuntu 18.04.1 LTS
  • Postfix 3.3.0

Install Postfix

I’m not going to go into a lot of explanation. I’m simply going to state the steps and configuration I did which got everything working.

First thing you need to do is install Postfix.

$ sudo apt-get update
$ sudo apt-get install postfix mailutils

NOTE For the Postfix installation, when it asks for the “System mail name” value, it should be the same as the name of the server.

Configure Yahoo! Account

The Yahoo! Account’s security setting must be set to allow password authentication. By default, it is not enabled. Go to the Yahoo! Account Security page and make sure “Password is enabled” is set as showing in Figure 1.

Figure 1 - Yahoo! Account Security

Yahoo! Account Security
Yahoo! Account Security

Configure Postfix Authentication for Yahoo!

Configure Postfix to login to Yahoo!’s SMTP server. This configures the server name, port, account id, and clear text password.

# Create the password file
$ cd /etc/postfix/sasl
$ touch sasl_passwd_yahoo
$ chmod 600 sasl_passwd_yahoo

Now edit the sasl_passwd_yahoo file using your favorite editor. Make it look like Listing 1, replacing ACCOUNT_NAME and CLEAR_TEXT_PASSWD appropriately.

Listing 1 - sasl_passwd_yahoo

[smtp.mail.yahoo.com]:465 ACCOUNT_NAME@yahoo.com:CLEAR_TEXT_PASSWD

Now hash the sasl_passwd_yahoo file into a Postfix .db file

$ postmap sasl_passwd_yahoo

Configure Postfix Email Mapping for Yahoo!

If user mike attempts to send an email from a server with the name bluegreensky, the default Postfix FROM address will be mike@bluegreensky. This is not good because if you try to relay this email through Yahoo!, it will be blocked since the FROM address does not match the Yahoo! account. To get around this, configure a regular expression mapping file that will change all local email addresses (like mike@bluegreensky) to the Yahoo! account email address.

First, create a generic map file that’s empty

$ mkdir /etc/postfix/map
$ cd /etc/postfix/map
$ touch generic_map
$ chmod 600 generic_map
# Hash the file into a *.db file
$ postmap /etc/postfix/map/generic_map

Next, create a yahoo map file that will change local email address to the Yahoo! account email address.

$ mkdir /etc/postfix/map
$ cd /etc/postfix/map
$ touch regex_map_yahoo 
$ chmod 600 regex_map_yahoo 

Now edit the regex_map_yahoo file using your favorite editor. Make it look like Listing 2, replacing HOSTNAME - just HOSTNAME, not @HOSTNAME…don’t lose the @ character - and ACCOUNT_NAME appropriately.

Listing 2 - regex_map_yahoo

/.+@HOSTNAME/    ACCOUNT_NAME@yahoo.com

Now hash the regex_map_yahoo file into a Postfix .db file

$ postmap regex_map_yahoo

Configure Postfix SSL for Yahoo!

SSL must be used to connect to Yahoo! SMTP servers. During Postfix installation, a main.cf is created. It must be edited

$ cd /etc/postfix

Now edit the main.cf file using your favorite editor. In Listing 3 are the values you must add or update in main.cf. For each of the name/value pairs below, search main.cf to see if it already exists. If so, use my value below. If not, add my value to the end of main.cf.

Listing 3 - main.cf

# Yahoo!
relayhost = [smtp.mail.yahoo.com]:465
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_security_options =
smtp_sasl_password_maps = hash:/etc/postfix/sasl/sasl_passwd_yahoo
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
smtp_tls_wrappermode = yes
smtp_tls_security_level = encrypt
smtp_generic_maps = hash:/etc/postfix/map/generic_map, regexp:/etc/postfix/map/regex_map_yahoo

Restart Postfix

Restart Postfix so it picks up all the new configuration.

bash $ service postfix restart

Testing Postfix

If all goes well, Postfix restarted without any errors and is now configured to relay email from the server through the Yahoo! account. I typically test this with both mail and at.

# Test sending mail directly
$ echo "test email message" | mail -s "test email from server" some_email_address@someprovider.com
# Test sending mail through scheduler at
$ echo "echo \"gosh golly, it is AT\"" | at now

Do this testing, and see if you get the email your are expecting.

NOTE When testing with at, you will want to create a ~/.forward file with your email address so the results of the scheduled job are not delivered to your server’s local account’s inbox.

Summary

This post shows how to configure Postfix to relay email through a Yahoo! account over SSL. Sending email from a home network and have it not be blocked is tough. This configuration got me working. I hope it works for you.

November 09, 2018

Derby Database Backup

Abstract

I have already posted a number of blogs about Derby:

This wasn’t intended to be a series. But over the years I’ve been using Derby more and more. Recently, I started using Derby as my database of choice for my Microservice architecture. These are personal-use applications, so Derby is more than sufficient. Even though these are personal-use applications, I require multiple servers with limited user permissions and - most importantly - backup. I’d hate to lose my data! The purpose of this post is to demonstrate how to backup a Derby database.

Disclaimer

This post is solely informative. Critically think before using any information presented. Learn from it but ultimately make your own decisions at your own risk.

Requirements

I did all of the work for this post using the following major technologies. You may be able to do the same thing with different technologies or versions, but no guarantees.

  • Apache Derby 10.14.2.0
  • OpenJDK 64-Bit Server VM Zulu11.1+23 (build 11-ea+22, mixed mode)

Download

There are no downloads with this blog post. The scripts are shown in full.

Derby System Utility

Backing up a Derby database is really quite simple. Derby has a built-in system utility for performing the backup. The utility is SYSCS_UTIL.SYSCS_BACKUP_DATABASE('/location/of/the/backup/'). When called, Derby will lock the database and perform the copy operation to the file system location you specify as the parameter to SYSCS_BACKUP_DATABASE. Now that we know the system utility to do the backup, let’s look at a bash script to automate it.

Backup Script

Listing 1 is a bash script which can be easily modified to backup any Derby database on any network server.

Listing 1 - derby-mydatabase-backup.sh

#!/bin/bash

# Define a bunch of variables which will be used within this script.
# The names of the variables should be self-explanatory.
DERBY_HOME=/opt/db-derby-10.14.2.0-bin/
NETWORK_SERVER_HOST=localhost
NETWORK_SERVER_PORT=1527
DATABASE_NAME=mydatabase
DATABASE_USER=sa
DATABASE_PASSWORD=abc123
JDBC_URL="jdbc:derby://$NETWORK_SERVER_HOST:$NETWORK_SERVER_PORT/$DATABASE_NAME"
BACKUP_DIRECTORY="/tmp/$DATABASE_NAME-backup/$NETWORK_SERVER_PORT"
BACKUP_SCRIPT="$BACKUP_DIRECTORY/backup.sql"

# Remove old backup if it exists. It is not a good idea to
# perform a backup on top of an existing backup.
rm -rf $BACKUP_DIRECTORY
mkdir -p $BACKUP_DIRECTORY
cd $BACKUP_DIRECTORY

# Use the `echo` command to dynamically create an SQL file.
# This SQL file will be used by Derby `ij` to connect to
# the database and perform the backup.
echo "connect '$JDBC_URL' user '$DATABASE_USER' password '$DATABASE_PASSWORD';" >> $BACKUP_SCRIPT
echo "CALL SYSCS_UTIL.SYSCS_BACKUP_DATABASE('$BACKUP_DIRECTORY');" >> $BACKUP_SCRIPT
echo "exit;" >> $BACKUP_SCRIPT

# Run the Derby `ij` application, passing it the SQL file
# that was just dynamically created. `ij` will read the 
# SQL file, executing its commands. This will then
# cause `ij` to connect to the database and call the 
# system utility to perform the backup.
$DERBY_HOME/bin/ij $BACKUP_SCRIPT

Let’s take a look at this script in more detail.

Lines 5–15 setup a number of variables used within the script. Some variables are used to set the values of other variables. There is nothing too complicated here. The names of the variables are self-explanatory.

Lines 17–19 is file system maintenance. It is not a good idea to perform a backup on top of an existing backup. So these lines remove an existing backup (if it exists) and creates a new, empty, backup directory.

Lines 24–26 are then responsible for creating the backup.sql script file. This script file contains the SQL commands to perform the backup. Line 24 is the connect command so Derby ij can connect to the database you want to backup. Line 25 is where the magic happens with a call to the SYSCS_BACKUP_DATABASE system utility. The location of the backup directory is passed as a parameter to the utility. When this SQL command is executed, Derby will lock the database and perform the backup. Line 26 is the exit command to exit ij.

Line 33 is then finally where everything happens. The Derby ij command is called with the location of the dynamically created backup.sql file passed to ij as a command-line parameter. When bash executes line 33, and if everything goes well, the Derby database will be backed up.

NOTE If you are running the Derby network server with a Java security policy, you may run into some problems with this script. the Java SecurityManager may prevent the network connection to the database or the SecurityManager my encounter permission problems writing to the backup directory.

Summary

Backing up a Derby database is pretty easy. Just call SYSCS_UTIL.SYSCS_BACKUP_DATABASE('/location/of/the/backup/').

References

Backing Up a Database. (2013, January 24). db.apache.org. Retrieved from https://db.apache.org/derby/docs/10.0/manuals/admin/hubprnt43.html.